Table of Contents |
---|
Introduction
This is the second in a series of articles describing and describes how to get started with using the the JADE perform simple file selection with the YADE Client via its Command Line Interface.Articles in this series are:
YADE Tutorials List
YADE Client Command Line Interface
- Getting Started and Downloading files
- Simple File Selection
- More Advanced File Selection
- Checking files for completeness
- Public / Private Key Authentication
...
YADE Background Service
Prerequisites
See the Using the tutorials with the YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.
Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.
Download file
The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor from the following link:
Scope of this tutorial
Before starting to transfer files JADE YADE reads the contents of the folder specified using the source_dir one or more child elements of the Selection parameter.
In the examples in first article tutorial in this series a default setting, that all the files in the source directory will be downloaded, appliedwas used: .* .In this article the source directory file list will be filtered so that only specific files are transferred
The basic file selection methods will be described in this tutorial:
File Name Filtering and Recursive File
...
In the examples JADE was used to transfer all the files from the directory specified in the source_dir parameter.
Selection
The However, the following profile shows how JADE YADE can be used to:
- Filter the names of files to be transferred using a regular expression
- Transfer files from sub-directories recursively
Example
In the profile listed below, two new parameters have been added to the sftp_server_2_local_pass example from above:
Configuration structure
ProtocolFragment
The ftp_demo_sos-berlin ProtocolFragment used in the Client Command Line Interface - Getting Started tutorial will be reused.
Profile
The example presented in this tutorial uses a Profile based on the ftp_server_to_local Profile described in the Getting Started tutorial.
The new Profile, has been given the profile_id = ftp_server_to_local_select_recursive and the following changes have been made:
The value of the file_spec defines the regular expression used to filter file names
- recursive specifies whether the transfer should be carried out recursively A recursive child element has been added to the FileSpecSelection element and its value set to true (the default value is false).
- The source_dir parameter has also been changed to:
- /rec
The relevant file structure on the source server is:
/
/rec
test.txt
test_1.txt/sd
test_2.txt
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini
formats.
Code Block | ||||
---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Fragments>
<ProtocolFragments>
<FTPFragment name="ftp_demo_sos-berlin">
<BasicConnection>
<Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
</BasicConnection>
<BasicAuthentication>
<Account><![CDATA[demo]]></Account>
<Password><![CDATA[demo]]></Password>
</BasicAuthentication>
</FTPFragment>
</ProtocolFragments>
</Fragments>
<Profiles>
<Profile profile_id="ftp | ||||
Code Block | ||||
| ||||
[sftp_server_2_local_select_recursive] ;; operation"> <Operation> <Copy> <CopySource> <CopySourceFragmentRef> <FTPFragmentRef ref= copy file_spec"ftp_demo_sos-berlin" /> </CopySourceFragmentRef> <SourceFileOptions> <Selection> <FileSpecSelection> = <FileSpec><![CDATA[^test_.\.txt$]]></FileSpec> recursive <Directory><![CDATA[./rec]]></Directory> <Recursive>true</Recursive> </FileSpecSelection> </Selection> </SourceFileOptions> </CopySource> <CopyTarget> <CopyTargetFragmentRef> <LocalTarget /> = true ;;force_files</CopyTargetFragmentRef> <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory> </CopyTarget> </Copy> = </Operation> </Profile> </Profiles> </Configurations> |
Code Block | ||||
---|---|---|---|---|
| ||||
[protocol_fragment_ftp@ftp_demo_sos-berlin] protocol = ftp host false ;; source_host = test.sos-berlin.com source_protocoluser = sftp source_ssh_auth_method = password source_portdemo password = demo [ftp_server_2_local_select_recursive] operation = 22copy source_userinclude = protocol_fragment_ftp@ftp_demo_sos-berlin source_passwordfile_spec = demo ^test_.\.txt$ source_dir = ./rec ;; target_hostrecursive = localhosttrue target_protocol = local target_dir = ${USERPROFILE}\jade_demo\a |
Call this profile by entering the following in the command line:
|
Running the Profile
This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:
Code Block | ||
---|---|---|
| ||
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_select_recursive" |
Code Block | ||||
---|---|---|---|---|
| ||||
Code Block | ||||
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_select_recursive" |
On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:
Code Block | ||
---|---|---|
| ||
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_select_recursive" |
Code Block | ||
---|---|---|
| ||
./jade.sh -settings.ini="${HOME}/jade_demo/sos-berlin_demo_2_local_select.xml" -profile="sftpftp_server_2_local_select_recursive" |
Behavior
...
The relevant file structure on the source server is:
/
/rec
test.txt
test_1.txt/sd
test_2.txt
- YADE JADE will download the test_1.txt from the /rec folder on the server and ignore the test.txt file whose name does not match with the file_spec FileSpec regular expression.
- JADE YADE will create a /sd folder in the target a directory if this folder does not already exist.
- It will the open the /rec/sd sub-folder on the server, create a corresponding folder on the target and download the test_2.txt file to this folder.
Behavior
...
with DisableErrorOnNoFilesFound = true
As mentioned above, the default value for the force_file parameter is true, meaning that even if this parameter is not specified, the JADE engine will throw an error if no match is found.
The default behavior of YADE is to throw an error if a file transfer is attempted and no files are found. The DisableErrorOnNoFilesFound element is used to allow YADE to attempt a file transfer without an error being generated if no files are found. This behavior can be useful when, for example, polling for files.
The DisableErrorOnNoFilesFound element is specified as a Directives child element, the next screenshot shows how this element is integrated in a configuration:
The effect of setting the DisableErrorOnNoFilesFound element can be demonstrated by, for example, setting the FileSpec regular expression in the Profile element to This can be demonstrated by changing the file_spec regular expression in the profile to, for example ^text\.txt$
and recalling the profileoperation.
The file transfer will now end with an Exit Code 99.
If, however, the force_file = true parameter in the example profile is commented in, JADE the DisableErrorOnNoFilesFound element is set to true, YADE will not throw an error.
Note:
- The meaning of the values true and false for the DisableErrorOnNoFilesFound element are the opposite to the meaning for the element's
settings.ini
file predecessor, force_files.
Zero-Byte Files
The JADE YADE Client can filter handle files with zero bytes using the zero_byte_transfer parameter.
Example
in a number of ways, depending on the setting used for the TransferZeroByteFiles parameter, which is specified as a child of the Directives element.
Test Example
The default TransferZeroByteFiles setting is true. The screenshot below shows how this element is integrated in a duplicate of the ftp_server_2_local file transfer profile that was described in the first tutorial in this series. This new Profile has profile_id = ftp_server_2_local_zero_byte.
In its original configuration - i.e. with FileSpec set to .*, the ftp_server_2_local profile transfers six files from the SOS test server to the local file system. One of these files - ./test_4.txt - The test_4.txt file on our test server has zero bytes and can be used to demonstrate the use of the zero_byte_transfer TransferZeroByteFiles parameter.
Note that alternative values of the file_spec and zero_byte_transfer parameters are included in the example profile but have been commented out (using ;;). These alternative values allow the functioning of the zero_byte_transfer parameter to be properly demonstrated. The default zero_byte_transfer setting is yes.
The screenshot also shows the relevant extract from the parameter reference for the TransferZeroByteFiles element with a description of the meaning of the different values the TransferZeroByteFiles element can have.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini
formats.
Code Block | ||||
---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="utf-8"?>
<Configurations xsi:noNamespaceSchemaLocation="http://www.sos-berlin.com/schema/yade/YADE_configuration_v1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Fragments>
<ProtocolFragments>
<FTPFragment name="ftp_demo_sos-berlin">
<BasicConnection>
<Hostname><![CDATA[test.sos-berlin.com]]></Hostname>
</BasicConnection>
<BasicAuthentication>
<Account><![CDATA[demo]]></Account>
<Password><![CDATA[demo]]></Password>
</BasicAuthentication>
</FTPFragment>
</ProtocolFragments>
</Fragments>
<Profiles>
<Profile profile_id="ftp | ||||
Code Block | ||||
| ||||
[sftp_server_2_local_zero_byte] ;; operation"> <Operation> <Copy> <CopySource> <CopySourceFragmentRef> <FTPFragmentRef ref= copy file_spec"ftp_demo_sos-berlin" /> </CopySourceFragmentRef> <SourceFileOptions> <Selection> <FileSpecSelection> = ^test_[0-9]\.txt$ ;;file_spec <FileSpec><![CDATA[.*]]></FileSpec> <Directory><![CDATA[./]]></Directory> </FileSpecSelection> </Selection> <Directives> <DisableErrorOnNoFilesFound>true</DisableErrorOnNoFilesFound> = ^test_[4]\.txt$ ;;zero_byte_transfer <TransferZeroByteFiles><![CDATA[false]]></TransferZeroByteFiles> </Directives> </SourceFileOptions> </CopySource> <CopyTarget> <CopyTargetFragmentRef> <LocalTarget /> </CopyTargetFragmentRef> <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory> </CopyTarget> = no zero_byte_transfer</Copy> </Operation> </Profile> = </Profiles> </Configurations> |
Code Block | ||||
---|---|---|---|---|
| ||||
[protocol_fragment_ftp@ftp_demo_sos-berlin] protocol relaxed ;;zero_byte_transfer = strictftp ;; source_host host = test.sos-berlin.com source_protocoluser = sftp source_ssh_auth_method = password source_portdemo password = demo [ftp_server_2_local_zero_byte] operation = 22copy source_userinclude = protocol_fragment_ftp@ftp_demo_sos-berlin source_passwordfile_spec = demo.* source_dir = / ;; target_host./ force_files = false zero_byte_transfer = localhostfalse target_protocol = local target_dir = ${USERPROFILE}\jade_demo\a |
...
Running the Profile
This profile is called on Windows systems using one of the following commands, depending on the YADE version being used:
Code Block | ||||
---|---|---|---|---|
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\jade_settingssos-berlin_demo_2_local_select.ini" -profile="sftpftp_server_2_local_zero_byte" |
Behavior
Code Block | ||
---|---|---|
| ||
jade.cmd -settings="%USERPROFILE%\jade_demo\sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_zero_byte" |
On Unix systems the profile is called using one of the following commands, depending on the YADE version being used:
Code Block | ||
---|---|---|
| ||
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.ini" -profile="ftp_server_2_local_zero_byte" |
Code Block | ||
---|---|---|
| ||
./jade.sh -settings="${HOME}/jade_demo/sos-berlin_demo_2_local_select.xml" -profile="ftp_server_2_local_zero_byte" |
Behavior
- With the TransferZeroByteFiles element set to false or relaxed (and with the FileSpec set to
^test_.\.txt$
as listed above) neither will With the zero_byte_transfer parameter set to relaxed (i.e. with the file_spec and zero_byte_transfer parameters set as listed above) the test_4.txt zero byte file be downloaded nor will not an error be downloaded.raised. - Setting the TransferZeroByteFiles element to false and rerunning the example A setting of zero_byte_transfer = no will:
- cause Cause the the test_4.txt file to be ignored as with the relaxed setting and
- cause Cause an error to be raised if no files with more than zero bytes are found - i.e. if no files are to be transferred.
This can be demonstrated by activating the alternative file_spec regex (changing the FileSpec regex to^test_[4]\.txt$
) which will only match for the zero byte file.
- A setting of zero_byte_transfer of TransferZeroByteFiles = strict will cause an error to be raised if any zero byte files are matched.