Table of Contents |
---|
Introduction
This is the third in a series of articles describing how to get started with using the the JADE YADE Client via its Command Line Interface .The articles in this series are:and covers:
- Polling a Source Directory
YADE Tutorials List
YADE Client Command Line Interface
- Getting Started and Downloading files
- Simple File Selection
- More Advanced File Selection
- Checking files for completeness
- Public / Private Key Authentication
...
YADE Background Service
Prerequisites
See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.
Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.
Download file
The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor using the following link:
Polling a source directory for a minimum number of files
JADE YADE can poll a source directory at regular intervals until it finds a file name match or until a specified length of time has elapsed. If it finds a file name match with one or more files it will then transfer the matching file(s) and then stop polling.
JADE YADE can also delay transfer until a minimum number of file name matches has been found.
JADE YADE will transfer all those files it has found at the end of the polling interval if the minimum number of matches has not been reached.
Example
Configuration structure
ProtocolFragment
The ftp_demo_sos-berlin ProtocolFragment used in the first tutorial in this series will be reused.
Profile
The example presented in this tutorial uses a Profile based on the ftp_server_to_local Profile described in the first tutorial in this series.
The new Profile, has been given the profile_id = ftp_server_to_local_poll and the following changes have been made:
FileSpec
The value of the FileSpec element was changed to ^test_.\.txt$
. Five files will be found in the transfer source directory.
Polling Elements
The Polling parent element is added as a child of the SourceFileOptions element. In the example profile listed shown in the XML Editor screenshot below the function of the four poll_* parameters three polling parameters should be self-explanatory but the following should be noted:
poll_interval PollInterval - is specified in seconds (only integer values)
poll_timeout PollTimeout - is specified in minutes (only integer values)
poll_minfiles MinFiles - optional - can only be used together with poll_timeout PollTimeout
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini
formats.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| [sftp
| |||||||
<?xml version="1.0" encoding="utf-8"?> <Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <Fragments> <ProtocolFragments> <FTPFragment name="ftp_demo_sos-berlin"> <BasicConnection> <Hostname><![CDATA[test.sos-berlin.com]]></Hostname> </BasicConnection> <BasicAuthentication> <Account><![CDATA[demo]] ;; operation></Account> <Password><![CDATA[demo]]></Password> </BasicAuthentication> </FTPFragment> </ProtocolFragments> </Fragments> <Profiles> <Profile profile_id= copy file_spec"ftp_server_2_local_poll_minfiles"> <Operation> <Copy> <CopySource> <CopySourceFragmentRef> <FTPFragmentRef ref="ftp_demo_sos-berlin" /> </CopySourceFragmentRef> <SourceFileOptions> <Selection> <FileSpecSelection> = <FileSpec><![CDATA[^test_[0-9]\.txt$ ;; poll_interval .\.txt$]]></FileSpec> <Directory><![CDATA[./]]></Directory> </FileSpecSelection> </Selection> <Polling> <PollInterval>20</PollInterval> <PollTimeout>1</PollTimeout> <MinFiles>3</MinFiles> </Polling> </SourceFileOptions> </CopySource> <CopyTarget> = 20<CopyTargetFragmentRef> poll_timeout <LocalTarget /> = 1 poll_minfiles</CopyTargetFragmentRef> <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory> </CopyTarget> </Copy> </Operation> = </Profile> </Profiles> </Configurations> |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
[protocol_fragment_ftp@ftp_demo_sos-berlin] protocol 3 poll_keep_connection = trueftp ;; source_hosthost = test.sos-berlin.com source_protocoluser = sftp source_ssh_auth_methoddemo password = demo [ftp_server_2_local_poll_minfiles] operation = password source_port = 22copy source_userinclude = protocol_fragment_ftp@ftp_demo_sos-berlin source_passwordfile_spec = demo ^test_.\.txt$ source_dir = / ;; target_host./ poll_interval = 20 poll_timeout = 1 poll_minfiles = localhost3 target_protocol = local target_dir = ${USERPROFILE}\jade_demo\a |
...
Running the Profile
This profile is called on Windows systems using one of the following commands, depending on the YADE version being used profile by entering the following in the Windows command line:
Code Block | ||||
---|---|---|---|---|
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_poll.ini" -profile="ftp_server_2_local_poll_minfiles" |
Code Block | ||||
---|---|---|---|---|
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_poll.xml" -profile="ftp_server_2_local_poll_minfiles" |
On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:
Code Block | ||||
---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_poll.ini" -profile="sftpftp_server_2_local_poll_minfiles" |
Example behavior with poll_minfiles = 3
Code Block | ||||
---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_poll.xml" -profile="ftp_server_2_local_poll_minfiles" |
Example behavior with MinFiles less than the number of files found
- YADE runs the first polling operation immediately and JADE will transfer all the files found at the first polling - in this case the 5 test_*.txt files - as more than the minimum number of files has been found. Polling will then be stopped.
- If no files were to be found then the JADE YADE engine would throw an error as long as force_file DisableErrorOnNoFilesFound = true has not been set.
Example behavior with
...
MinFiles more than the number of files found
Change the poll_minfiles MinFiles parameter in your settings file the XML Editor to 6, save the settings file configuration and rerun the example.
- JADE YADE will find the five test_*.txt files when it polls but will not transfer the files as the minimum number of 6 has not been reached.
- JADE YADE will repeat polling until the time set in the poll_timeout PollTimeout parameter (in with the example configuration after 1 minute 20 secs) has expired.
- Note that all the files that have been found will be transferred once the timeout has been reached.
...