...
This is the fifth in a series of articles describing how to get started with using the JADE Client via its Command Line Interface and covers three methods of checking for file completnesscompleteness.
JADE Tutorials List
JADE Client Command Line Interface
...
The configurations described in this tutorial can be downloaded and then directly opened in the XML Editor. Links for each configuration are available at the start of each section of this tutorial.
...
Scope of this Tutorial
A number of methods for checking whether file transfer has been completed are presented in this document:
...
Use of the transactional parameter is generally recommended if more than one file is to be transferred.
File Completeness - Integrity Hash Checksums
Download file
The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:
...
Example a) - Creating an integrity hash file
JADE can generate an MD5 checksum file for a file. This file which is given the ending .md5 andcan then be transferred along with the 'original' file. Note however that JADE only generates the checksum file during a file transfer operation, whilst it is reading the 'original' file from the transfer source. This means that this feature has a limited scope - in practice JADE is most often used to check files for completeness using integrity hash files that have been generated by other sources.
...
- This example uses a set of 5 test_*.txt files that have been saved to the user's local file system during the implementation of an earlier tutorial in this series - for example The JADE Client Command Line Interface - Tutorial 1 - Getting Started in which the files were downloaded to the user's
\jade_demo\adirectory. - A ProtocolFragment is not required in the configuration of this example, as local to local transfer is being carried out.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
[local_2_local_create_md5]
operation = copy
source_protocol = local
file_spec = ^(test)(_)[0-9]\.txt
source_dir = ${USERPROFILE}\jade_demo\a
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
check_security_hash = true
create_security_hash_file = true |
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_create_md5" |
Behavior
If you have already run some of the examples described in previous tutorials, you will have a series of files in the ${USERPROFILE}\jade_demo\a directory on your computer, sequentially named test_1.txt through to test_5.txt.
...
The next example shows the use of the integrity hash files to verify files that have been transferred.
Example b) - Checking transferred files against transferred integrity hash files
After you have run the Create and transfer an MD5 checksum file example above you will have a series of files in your target ${USERPROFILE}\jade_demo\b directory together with the corresponding hash files.
...
- first as described below and then
- after modifying the content of the 'original' files to create an integrity hash mismatch.
XML Editor Configuration
The local_2_local_check_md5 profile shown in the screenshot below is almost identical to the previous local_2_local_create_md5 profile - the only significant difference is that it does not have the CreateIntegrityHashFile parameter as the files are already there.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
[local_2_local_create_md5]
operation = copy
source_protocol = local
file_spec = ^(test)(_)[0-9]\.txt
source_dir = ${USERPROFILE}\jade_demo\a
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b
check_security_hash = true
create_security_hash_file = true
[local_2_local_check_md5]
operation = copy
source_protocol = local
file_spec = ^(test)(_)[0-9]\.txt
source_dir = ${USERPROFILE}\jade_demo\b
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b\checked
check_security_hash = true |
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_file_complete.xml" -profile="local_2_local_check_md5" |
Behavior
First the 'original' and then the MD5 files will be copied to the b\checked sub-folder.
...
- Files that have already been checked and fully transferred will only be deleted from the target directory - in this case the checked directory - if transactional transfer has been specified.
- File transfers that have not been completed / started when the error occurs will be aborted / rolled back.
File Completeness - Atomic File Transfer
Example - Atomic File Transfer
The principle of atomic file transfer was described at the beginning of this article.
...
XML Editor Configuration
The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol = ftp
host = test.sos-berlin.com
user = demo
password = demo
[ftp_server_2_local_atomic]
operation = copy
source_include = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec = ^test_large_.\.txt$
source_dir = ./large
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\a\large
atomic_suffix = ~ |
Running the Atomic Transfer Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic" |
Behavior - Atomic File Transfer
When atomic file transfer is specified JADE will add the characters specified in the AtomicPrefix and AtomicSuffix parameters to the file name. Once the file transfer has been completed, JADE deletes the prefix and suffix.
...
| Code Block | ||
|---|---|---|
| ||
main INFO 09:09:18,622 (SOSDataExchangeEngine.java:1050) ::setInfo 3 files found for regexp '^test_large_.\.txt$'. main INFO 09:09:19,388 (SOSVfsLocalFile.java:413) ::rename SOSVfs_I_150: Datei 'C:\Users\aa\jade_demo\a\large\test_la rge_1.txt~' umbenannt in 'C:\Users\aa\jade_demo\a\large\test_large_1.txt'. |
File Completeness - Check Steady State
Example
...
Description
The local_2_local_check_steady_state profile listed below is intended to be used in JADE together with a second file transfer program that allows the file transfer rate to be restricted. This second file transfer program is used to transfer a series of files from our test server at a relatively slow rate to the local jade_demo/a folder used in previous examples. In parallel, JADE is used to check the size of files arriving in the folder jade_demo/a. JADE will transfer files to the target folder jade_demo/b once a steady state for each file has been found.
...
- Prepare your command line interface to run JADE with the local_2_local_check_steady_state profile using the command line call listed below but do not yet start the transfer.
- The second file transfer program should be configured to allow a maximum rate of around 16 kB/s. This transfer rate will mean that the download of each example file (approx. 220kB ) will take around 15 seconds. This will provide sufficient time for JADE to be started with the local_2_local_check_steady_stateProfile below, find the files being transferred and note that its own file transfer operation cannot be started until the download has been completed.
- Log onto our test FTP server with the second file transfer program using the same host, user & password as listed in other examples on this page.
- Select the three test_large_*.txt files, which you will find in the demo user's
largefolder.- Start downloading the selected files to the jade_demo/a folder.
- Quickly start JADE with the local_2_local_check_steady_state' profile.
Download file
The configuration used for this example can be downloaded using the following link and then directly opened in the XML Editor:
XML Editor Configuration
Local to local file transfer is used in the JADE part of this example so a ProtocolFragment element is not required. The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
[local_2_local_steady_state]
operation = copy
source_protocol = local
file_spec = ^test_large_.\.txt$
source_dir = ${USERPROFILE}\jade_demo\a
check_steady_state_of_files = true
check_steady_state_interval = 1
steady_state_count = 30
target_protocol = local
target_dir = ${USERPROFILE}\jade_demo\b |
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE version being used:
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/local_2_local_steady_state.xml" -profile="local_2_local_steady_state"
./jade.sh -settings="${HOME}/jade_demo/local_2_local_steady_state.xml" -profile="local_2_local_steady_state" |
Behavior
Start the profile using:
| Code Block | ||
|---|---|---|
| ||
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="local_2_local_check_steady_state" |
Behavior - Check Steady State of incoming files
Once JADE starts executing the profile the command line interface will show a series of messages containing the following:
...