Page History

Table of Contents

Introduction

This is the third in a series of articles describing how to get started with using the the JADE YADE Client via its Command Line Interface .Articles in this series are:and covers:

• Polling a Source Directory

YADE Tutorials List

YADE Client Command Line Interface

1. Getting Started and Downloading files
2. Simple File Selection
3. More Advanced File Selection

4. File Transfer

5. Checking files for completeness
6. Public / Private Key Authentication

...

2. Transfer via a Jump Host / DMZ

YADE Background Service

1. Installation, Configuration and Use

Prerequisites

See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.

Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.

Download file

The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor using the following link:

• sos-berlin_demo_2_local_poll.xml

Polling a source directory for a minimum number of files

JADE YADE can poll a source directory at regular intervals until it finds a file name match or until a specified length of time has elapsed. If it finds a file name match with one or more files it will then transfer the matching file(s) and then stop polling.

JADE YADE can also delay transfer until a minimum number of file name matches has been found.

JADE YADE will transfer all those files it has found at the end of the polling interval if the minimum number of matches has not been reached.

Example

Configuration structure

ProtocolFragment

The ftp_demo_sos-berlin ProtocolFragment used in the first tutorial in this series will be reused. 

Profile

The example presented in this tutorial uses a Profile based on the ftp_server_to_local Profile described in the first tutorial in this series.

The new Profile, has been given the profile_id = ftp_server_to_local_poll and the following changes have been made:

FileSpec

The value of the FileSpec element was changed to ^test_.\.txt$. Five files will be found in the transfer source directory.

Polling Elements

The Polling parent element is added as a child of the SourceFileOptions element. In the example profile shown in the XML Editor screenshot In the example profile listed below the function of the four poll_* parameters three polling parameters should be self-explanatory but the following should be noted:

• poll_interval PollInterval - is specified in seconds (only integer values)

• poll_timeout PollTimeout - is specified in minutes (only integer values)

• poll_minfiles MinFiles - optional - can only be used together with poll_timeout PollTimeout

XML Editor Configuration

Image Added

The Profile Code

The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini formats.

<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Fragments>
]
;;
 operation         <ProtocolFragments>
      <FTPFragment  = copy
 file_specname="ftp_demo_sos-berlin">
        <BasicConnection>
          = ^test_[0-9]\.txt$
 poll_interval<Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
        </BasicConnection>
       = 20<BasicAuthentication>
 poll_timeout          <Account><![CDATA[demo]]></Account>
         = 1
 poll_minfiles <Password><![CDATA[demo]]></Password>
        </BasicAuthentication>
      = 3</FTPFragment>
 poll_keep_connection   </ProtocolFragments>
  </Fragments>
 = true<Profiles>
;;
 source_host   <Profile profile_id="ftp_server_2_local_poll_minfiles">
       <Operation>
    = test.sos-berlin.com
 source_protocol  <Copy>
         = sftp<CopySource>
 source_ssh_auth_method    = password
 source_port     <CopySourceFragmentRef>
          = 22
 source_user  <FTPFragmentRef ref="ftp_demo_sos-berlin" />
           = demo
 source_password</CopySourceFragmentRef>
           = demo<SourceFileOptions>
 source_dir             <Selection>
   = /
;;
 target_host           <FileSpecSelection>
    = localhost
 target_protocol           = local
 target_dir <FileSpec><![CDATA[^test_.\.txt$]]></FileSpec>
                 = ${USERPROFILE}\jade_demo\a

Then call the profile by entering the following in the Windows command line:

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="sftp_server_2_local_poll_minfiles"

Example behavior with poll_minfiles = 3  

• JADE will transfer all the files found at the first polling - in this case the 5 test_*.txt files - as more than the minimum number of files has been found. Polling will then be stopped.
• If no files were to be found then the JADE engine would throw an error as long as force_file = true has not been set.

Example behavior with poll_minfiles = 6

Change the poll_minfiles parameter in your settings file to 6, save the settings file and rerun the example.

• JADE will find the five test_*.txt files when it polls but will not transfer the files as the minimum number of 6 has not been reached.
• JADE will repeat polling until the time set in the poll_timeout parameter (in the example 1 minute) has expired.
• Note that all the files that have been found will be transferred once the timeout has been reached.

File Transfer Examples

Renaming files

The following example combines two renaming possibilities:

• Renaming files
• Adding a date-stamp to file names

Example

The file name part(s) to be replaced is/are defined in the replacing parameter with round brackets () and the new part(s) in the replacement parameter, separated by semi-colons.

The example shows a two-part substitution:

• test will be replaced with file_ and
• '_' with the current date in the format specified.

[replace_local_datestamp]
;;
 operation <Directory><![CDATA[./]]></Directory>
                </FileSpecSelection>
     = copy
 file_spec       </Selection>
          = ^test_[0-9]\.txt$
 replacing  <Polling>
               = ^(test)(_)[0-9]\.txt
 replacement <PollInterval>20</PollInterval>
                = file_;[date:yyyy-MM-dd]_
;;
 source_host<PollTimeout>1</PollTimeout>
                = localhost
 source_protocol<MinFiles>3</MinFiles>
             = local</Polling>
   source_dir         </SourceFileOptions>
       = ${USERPROFILE}\jade_demo\a
;;
 target_host   </CopySource>
          <CopyTarget>
           = localhost<CopyTargetFragmentRef>
 target_protocol             =<LocalTarget local/>
 target_dir           </CopyTargetFragmentRef>
           = <Directory><![CDATA[${USERPROFILE}\jade_demo\b

Call the 'replace_local_datestamp' profile by entering the following in the command line:

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="replace_local_datestamp"

Behavior

The five test_*.txt files in the the a folder will be copied to the b folder and parts of their names will be replaced as described above.

File cumulation

JADE can cumulate the contents of individual files to a single target file:

• Files matching the file_spec regular expression are cumulated together into a new file
• The original files can be deleted if required
• There is not a de-cumulate parameter available at the moment

Example

The example profile downloads the five text_*.txt files from our demo server and adds their contents successively to a cumulative file.

The cumulative file is then given the name specified in the cumulative_file_name parameter in the profile.

The text specified in the  cumulative_file_separator parameter is incorporated in the cumulative file between the contents of the individual files.

The order in which the files are added to the cumulative file is not fixed but depends on the order in which the files arrive on the local file system.

[sftp_server_2_local_cumulate]
;;
 operationa]]></Directory>
          </CopyTarget>
        = copy</Copy>
 file_spec     </Operation>
    </Profile>
  </Profiles>
</Configurations>

[protocol_fragment_ftp@ftp_demo_sos-berlin]
protocol         = ^test_.\.txt$
 cumulate_files            = true
 cumulative_file_name      = text-files.txt
 cumulative_file_separator = --- File: %{SourceFileName} ---
;;cumulative_file_delete   = trueftp
;;
 source_host               = test.sos-berlin.com
 source_protocol           = sftp
 source_ssh_auth_method    = password
 source_porttest.sos-berlin.com
user                = 22
 source_user               = demo
 source_password           = demo
 source_dir                = /demo
;;
 target_host
[ftp_server_2_local_poll_minfiles]
operation               = localhost
 target_protocol           = local
 target_dircopy

source_include                    = ${USERPROFILE}\jade_demo\a

Run the example using:

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settings.ini" -profile="sftp_server_2_local_cumulate"

Behavior

Opening the cumulative file text-files.txt in a text editor after transfer has been completed shows the order in which individual files were added to the cumulative file.

Note that this will not necessarily follow the order in which transfer of individual files was started.

File compression and decompression

JADE can compress individual files locally using a gzip-compatible compression. Compressed files can then be stored locally or forwarded to a further destination. JADE is not currently able to include a number of files in an unpackable container as is usually done with desktop zip archivers.

Example

The example profile will cause JADE to download the two PDF files on the remote host, compress them and then store them in the local directory a.

The files have the following sizes before compression:

• lorem_ipsum_large.pdf - 115 KB
• lorem_ipsum_medium.pdf - 32 KB

[sftp_server_2_local_compress]
;;
 operation protocol_fragment_ftp@ftp_demo_sos-berlin
file_spec                  = copy
 file_spec                 = ^^test_.*\.pdf$txt$
 compresssource_filesdir            = true
;;
 source_host               = test.sos-berlin.com
 source_protocol/
poll_interval           = sftp
 source_ssh_auth_method    = password
 source_port        = 20
poll_timeout       = 22
 source_user               = demo1
 sourcepoll_passwordminfiles           = demo
 source_dir          = 3

target_protocol     = /large/
;;
 target_host               = localhostlocal
 target_protocoldir           = local
 target_dir                = ${USERPROFILE}\jade_demo\a

Running the Profile

This profile is called on Windows systems using one of the following commands, depending on the YADE version being usedRun the example using:

jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settingssos-berlin_demo_2_local_poll.ini" -profile="sftpftp_server_2_local_compress"

Behavior

After transfer and compression the files are stored with the default suffix .gz. This suffix can be changed if required by setting a compressed_file_extension parameter.

The example files will be reduced in size to approximately 26 KB (lorem_ipsum_large.pdf) and 23 KB (lorem_ipsum_medium.pdf).

File decompression

The decompress_after_transfer parameter can be used to decompress files after transfer.

poll_minfiles"

jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_poll.xml" -profile="ftp_server_2_local_poll_minfiles"

On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_poll.ini" -profile="ftp_server_2_local_poll_minfiles"

./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_poll.xml" -profile="ftp_server_2_local_poll_minfiles"

Example behavior with MinFiles less than the number of files found 

• YADE runs the first polling operation immediately and will transfer all the files found - in this case the 5 test_*.txt files - as more than the minimum number of files has been found. Polling will then be stopped.
• If no files were to be found then the YADE engine would throw an error as long as DisableErrorOnNoFilesFound = true has not been set.

Example behavior with MinFiles more than the number of files found

Change the MinFiles parameter in the XML Editor to 6, save the configuration and rerun the example.

• YADE will find the five test_*.txt files when it polls but will not transfer the files as the minimum number of 6 has not been reached.
• YADE will repeat polling until the time set in the PollTimeout parameter (with the example configuration after 1 minute 20 secs) has expired.
• Note that all the files that have been found will be transferred once the timeout has been reached.

 Alternatively, a program such as 7-Zip can be used to manually extract the original file from the archive.