Introduction
This is the fifth in a series of articles describing how to get started with using the JADE Client via its Command Line Interface and covers three methods of checking for file completness.
JADE Tutorials List
JADE Client Command Line Interface
- Getting Started and Downloading files
- Simple File Selection
- More Advanced File Selection
- Checking files for completeness
- Public / Private Key Authentication
- Transfer via a Jump Host / DMZ
JADE Background Service
Prerequisites
See the Using the tutorials with the JADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.
Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.
Download files
The configurations described in this tutorial can be downloaded and then directly opened in the XML Editor using the following links:
- Checking Files for
- local_2_local_file_complete.xml
Checking files for completeness
A number of methods for checking whether file transfer has been completed are presented in this document:
- Integrity hash checksums such as MD5, which are generated before file transfer starts and can be used to check files after transfer.
- Atomic file transfer, which is used to hide files being transferred from target directory monitors until the file transfer has been completed.
- A prefix and/or suffix is added to the name of the file for the duration of the transfer.
This prefix/suffix is chosen so that the file being transferred will not be matched by any target host file monitor. - Once transfer has been completed, JADE removes the prefix/suffix and the file becomes visible to a file monitor.
- A prefix and/or suffix is added to the name of the file for the duration of the transfer.
- Regularly checking the file size until it reaches a steady state. Here it is assumed that the transfer has been completed when two checks in series return the same size.
Other methods are:
- Repeating the file transfer and comparing the sizes of the file transfers. When both transfers have the same size then it is assumed that the transfer is complete.
See the CheckSize element documentation for more information. - Checking file size after transfer with the CheckSize element.
Use of the transactional parameter is generally recommended if more than one file is to be transferred.
Download file
The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:
File Completeness - Integrity Hash Checksums
Example a) - Creating an integrity hash file
JADE can generate an MD5 checksum file for a file. This file which is given the ending .md5 and can then be transferred along with the 'original' file. Note however that JADE only generates the checksum file during a file transfer operation, whilst it is reading the 'original' file from the transfer source. This means that this feature has a limited scope - in practice JADE is most often used to check files for completeness using integrity hash files that have been generated by other sources.
Note that to create and save an integrity hash, JADE requires write permissions for the target directory. This means that this feature cannot be demonstrated by downloading files from our test server, where the demo user only has read permissions. Instead it will be demonstrated through local transfer.
Note
- This example uses a set of 5 test_*.txt files that have been saved into the user's \jade_demo\a directory by an earlier tutorial in this series - for example The JADE Client Command Line Interface - Tutorial 1 - Getting Started .
- A ProtocolFragment is not required in the configuration of this example, as local to local transfer is being carried out.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini
formats.
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_create_md5" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5" ./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
Behavior
If you have already run some of the previous examples, you will have a series of files in the ${USERPROFILE}\jade_demo\a
directory on your computer, sequentially named test_1.txt through to test_5.txt.
JADE will carry out the following steps when executing the local_2_local_create_md5 Profile:
- Read each of the files in the source directory matching the FileSpec regular expression into memory
- Generate an integrity hash file for each 'original' files it finds matching the FileSpec.
- Save the integrity hash files along with the 'original' files in the transfer target directory (
${USERPROFILE}\jade_demo\b
). - An MD5 file corresponding to a file test_1.txt would carry the name test_1.txt.md5.
main INFO 13:57:35,512 (SOSFileListEntry.java:302) ::createChecksumFile SOSVfs_I_285: Checksum-Datei erzeugt: 'C:\Use rs\aa\jade_demo\b\test_1.txt.md5'
The next example shows the use of the integrity hash files to verify files that have been transferred.
Example b) - Checking transferred files against transferred integrity hash files
After you have run the Create and transfer an MD5 checksum file example above you will have a series of files in your target ${USERPROFILE}\jade_demo\b
directory together with the corresponding hash files.
When executing the profile in this example JADE will verify the 'original' files that were transferred to the b directory using their integrity hash files before writing the 'original' files to a new target directory ${USERPROFILE}\jade_demo\b\checked
.
To test whether the checks actually work, we suggest running the check twice:
- first as described below and then
- after modifying the content of the 'original' files to create an integrity hash mismatch.
XML Editor Configuration
The local_2_local_check_md5 profile shown in the screenshot below is almost identical to the previous local_2_local_create_md5 profile - the only significant difference is that it does not have the CreateIntegrityHashFile parameter as the files are already there.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini
formats.
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5" ./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
Behavior
First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.
If all of the 'original' files in the source directory b are intact (i.e. if you do not manipulate any of the files) then:
- a new integrity hash will be calculated for each of the 'original' files.
- These new hashes will then checked against those saved in the .md5 files.
If all the hashes are OK then:
- No information about the results of the integrity checks will be output.
- See Issue
Behavior in the Event of an Integrity Hash Mismatch
If any of the 'original' files in the source directory b has been corrupted (or if you have manipulated one or more of the files as suggested above) then an error will be thrown:
main ERROR 14:11:13,763 (SOSFileListEntry.java:1027) ::run SOSVfs_E_229: Fehler. Daten³bertragung nicht m÷glich. Grund: com.sos.JSHelper.Exceptions.JobSchedulerException: Integrity Hash violation. File C:\Users\aa\jade_demo\b\test_3.txt.md5, checksum read: 'b4817065c96c3103e11ab8295a8e0c2c', checksum calculated: 'dbeb552a55aad26cabf17641104bf64f' main ERROR 14:11:13,765 (SOSDataExchangeEngine.java:1275) ::transfer SOSDataExchangeEngine.TRANSFER_ABORTED main INFO 14:11:13,766 (SOSFileList.java:506) ::Rollback Rollback aborted files.
Note that:
- Files that have already been checked and fully transferred will only be deleted from the target directory - in this case the checked directory - if transactional transfer has been specified.
- File transfers that have not been completed / started when the error occurs will be aborted / rolled back.
File Completeness - Atomic File Transfer
Example - Atomic File Transfer
The principle of atomic file transfer was described at the beginning of this article.
The local_2_local_atomic profile listed below is will carry out atomic file transfer from our test server to a local folder. Three relatively large files (each 200KB) will be downloaded and theses may be sufficiently large for you to see that they are first of all written to the local file system with the ~ suffix while the download is taking place and that the suffix is removed once the download has bee completed.
Download file
The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:
XML Editor Configuration
The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini
formats.
Running the Atomic Transfer Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic" jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic" ./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
Behavior - Atomic File Transfer
When atomic file transfer is specified JADE will add the characters specified in the AtomicPrefix and AtomicSuffix parameters to the file name. Once the file transfer has been completed, JADE deletes the prefix and suffix.
Whilst this procedure may happen too quickly to be observed in a graphic file manager, the following INFO entries should be able to be seen in the command line interface:
main INFO 09:09:18,622 (SOSDataExchangeEngine.java:1050) ::setInfo 3 files found for regexp '^test_large_.\.txt$'. main INFO 09:09:19,388 (SOSVfsLocalFile.java:413) ::rename SOSVfs_I_150: Datei 'C:\Users\aa\jade_demo\a\large\test_la rge_1.txt~' umbenannt in 'C:\Users\aa\jade_demo\a\large\test_large_1.txt'.
File Completeness - Check Steady State
Example - Check Steady State of incoming files
The local_2_local_check_steady_state profile listed below is intended to be used in JADE together with a second file transfer program that allows the file transfer rate to be restricted. This second file transfer program is used to transfer a series of files from our test server at a relatively slow rate to the local jade_demo/a folder used in previous examples. In parallel, JADE is used to check the size of files arriving in the folder jade_demo/a. JADE will transfer files to the target folder jade_demo/b once a steady state for each file has been found.
WinSCP is an example of a windows file transfer program that allows transfer rate restriction and was used to test this example.
Carry out the following steps to run the example:
- Prepare your command line interface to run JADE with the local_2_local_check_steady_state profile using the command line call listed below but do not yet start the transfer.
- The second file transfer program should be configured to allow a maximum rate of around 16 kB/s. This transfer rate will mean that the download of each example file (approx. 220kB ) will take around 15 seconds. This will provide sufficient time for JADE to be started with the local_2_local_check_steady_stateProfile below, find the files being transferred and note that its own file transfer operation cannot be started until the download has been completed.
- Log onto our test FTP server with the second file transfer program using the same host, user & password as listed in other examples on this page.
- Select the three test_large_*.txt files, which you will find in the demo user's
large
folder.- Start downloading the selected files to the jade_demo/a folder.
- Quickly start JADE with the local_2_local_check_steady_state' profile.
Download file
The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:
XML Editor Configuration
Local to local file transfer is used in the JADE part of this example so a ProtocolFragment element is not required. The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini
formats.
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_steady_state.xml" -profile="local_2_local_steady_state" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_steady_state.xml" -profile="local_2_local_steady_state"
On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:
./jade.sh -settings="${HOME}/jade_demo/local_2_local_steady_state.xml" -profile="local_2_local_steady_state" ./jade.sh -settings="${HOME}/jade_demo/local_2_local_steady_state.xml" -profile="local_2_local_steady_state"
Behavior
Start the profile using:
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="local_2_local_check_steady_state"
Behavior - Check Steady State of incoming files
Once JADE starts executing the profile the command line interface will show a series of messages containing the following:
main INFO 10:37:00,924 (SOSDataExchangeEngine.java:1050) ::setInfo 3 files found for regexp '^test_large_.\.txt$'. main INFO 10:37:00,925 (SOSDataExchangeEngine.java:1050) ::setInfo checking file(s) for steady state main INFO 10:37:00,926 (SOSDataExchangeEngine.java:130) ::checkSteadyStateOfFiles steady check (1 of 10). waiting 2s. main INFO 10:37:02,931 (SOSDataExchangeEngine.java:148) ::checkSteadyStateOfFiles steady check (1 of 10). Changed. file size: new = 131072 bytes, old = 98304 bytes. 'C:\Users\aa\jade_demo\a\test_large_1.txt'
The two checkSteadyStateOfFiles INFO messages will repeat until a steady state is noted for each file and the transfer of that file is started.