Page History

...

JADE can generate an MD5 checksum file for a file. This file which is given the ending .md5 andcan then be transferred along with the 'original' file. Note however that JADE only generates the checksum file during a file transfer operation, whilst it is reading the 'original' file from the transfer source. This means that this feature has a limited validity scope and that in practice JADE checks is most often used to check files for completeness using integrity hash files that have been generated by another sourceother sources.

Note that to create and save an integrity hash file before a file transfer operation, JADE requires write permissions for the source target directory. In turn, this means that this feature cannot be demonstrated by downloading files from our test server, where the demo user only has read permissions. Instead it will be demonstrated through local transfer.

...

When executing the profile in this example JADE will verify the 'original' files that were transferred to the b directory using their associated integrity hash files before writing the 'original' files to a new target directory ${USERPROFILE}\jade_demo\b\checked.
However, to To test whether the checks actually work,  you can open one of the transferred files and modify part of its content before running the profile belowwe suggest running the check twice:

• first as described below and then
• after modifying the content of the 'original' files to create an integrity hash mismatch.

XML Editor Configuration

The local_2_local_check_md5 profile shown in the screenshot below is almost identical to the previous local_2_local_create_md5 profile - the only significant difference is that it does not have the CreateIntegrityHashFile parameter as the files are already there.

...

The local_2_local_atomic profile listed below is will carry out atomic file transfer from our test server to a local folder. As the transfer is likely to occur within one or two seconds it is unlikely that it will be possible for the functioning of the atomic transfer (i.e the removal of atomic prefix and suffixes) to be observed.Three relatively large files (each 200KB) will be downloaded and theses may be sufficiently large for you to see that they are first of all written to the local file system with the ~ suffix while the download is taking place and that the suffix is removed once the download has bee completed.

XML Editor Configuration

The configuration used to demonstrate atomic file transfer is uses the ftp_demo_sos-berlin Protocol Fragment already used in several of the tutorials in this series. The two FileSpecSelection elements in the Profile point to three large files in a subfolder on the test server. A tilde (~) is used as an AtomicSuffix and an AtomicPrefix is not required.

Image Added

The Profile Code

The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations>
  <Fragments>
    <ProtocolFragments>
      <FTPFragment name="ftp_demo_sos-berlin">

[sftp_server_2_local_atomic]
;;
 operation                    = copy
 file_spec                    = ^test_large_1\.txt$
;;
 atomic_prefix                = ~
 atomic_suffix                = ~
;;
 source_host        <BasicConnection>
          = <Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
 source_protocol       </BasicConnection>
       = sftp<BasicAuthentication>
 source_ssh_auth_method        = password
 source_port <Account><![CDATA[demo]]></Account>
          <Password><![CDATA[demo]]></Password>
         = 22</BasicAuthentication>
 source_user     </FTPFragment>
    </ProtocolFragments>
  </Fragments>
  <Profiles>
    <Profile profile_id= demo
 source_password"ftp_server_2_local_atomic">
      <Operation>
        <Copy>
   = demo
 source_dir      <CopySource>
            <CopySourceFragmentRef>
              <FTPFragmentRef ref= /large/
;;
 target_host"ftp_demo_sos-berlin" />
            </CopySourceFragmentRef>
            <SourceFileOptions>
              <Selection>
                  = localhost
 target_protocol              = local
 target_dir                   = ${USERPROFILE}\jade_demo\a

The Atomic Transfer profile is started using:

  <FileSpecSelection>
                  <FileSpec><![CDATA[^test_large_.\.txt$]]></FileSpec>
                  <Directory><![CDATA[./large]]></Directory>
                </FileSpecSelection>
              </Selection>
            </SourceFileOptions>
          </CopySource>
          <CopyTarget>
            <CopyTargetFragmentRef>
              <LocalTarget />
            </CopyTargetFragmentRef>
            <Directory><![CDATA[${USERPROFILE}\jade_demo\a\large]]></Directory>
            <TargetFileOptions>
              <Atomicity>
                <AtomicSuffix><![CDATA[~]]></AtomicSuffix>
              </Atomicity>
            </TargetFileOptions>
          </CopyTarget>
        </Copy>
      </Operation>
    </Profile>
  </Profiles>
</Configurations>

[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol                            = ftp

host                                = test.sos-berlin.com
user                                = demo
password                            = demo

[ftp_server_2_local_atomic]
operation                           = copy

source_include                      = protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                           = ^test_large_.\.txt$
source_dir                          = ./large

target_protocol                     = local
target_dir                          = ${USERPROFILE}\jade_demo\a\large
atomic_suffix                       = ~

Running the Atomic Transfer Profile

This profile is called on Windows systems using the following command, depending on the JADE version being used:

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"

On Unix systems the profile is called using one of the following commands, again depending on the JADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp_server_2_local_atomic"
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_atomic.xml" -profile="ftp

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="sftp_server_2_local_atomic"

Behavior - Atomic File Transfer

When atomic file transfer is specified JADE will add the characters specified in the atomic_prefix and atomic_suffix AtomicPrefix and AtomicSuffix parameters to the file name. Once the file transfer has been completed, JADE deletes the prefix and suffix.

Whilst this procedure will likely may happen to too quickly to be observed in a graphic file manager, the following INFO entries should be able to be seen in the command line interface:

 20218 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileListEntry 
 - SOSVfs_I_0108: transfer of /test_large_1.txt started
25943 [main] INFO com.sos.VirtualFileSystem.DataElements.SOSFileListEntry 
 - SOSVfs_I_274: The security hash (MD5) of the test_large_1.txt file is 
40f38a5f677aa387a057677e6d78aaa6
25949 [main] INFO com.sos.VirtualFileSystem.local.SOSVfsLocalFile 
 -  09:09:18,622   (SOSDataExchangeEngine.java:1050) ::setInfo 3 files found for regexp '^test_large_.\.txt$'.
main INFO  09:09:19,388   (SOSVfsLocalFile.java:413) ::rename SOSVfs_I_150: FileDatei 'C:\Users\tempaa\jade_demo\a\large\~testtest_largela
rge_1.txt~' renamedumbenannt toin 
'C:\tempUsers\aa\jade_demo\a/\large\test_large_1.txt'.

File Completeness - Check Steady State

...