Table of Contents |
---|
Introduction
This is the fourth in a series of articles describing how to get started with using the the JADE YADE Client via its Command Line Interface and covers:
...
YADE Tutorials List
...
YADE Client Command Line Interface
- Getting Started and Downloading files
- Simple File Selection
- More Advanced File Selection
- Checking files for completeness
- Public / Private Key Authentication
- Transfer via a Jump Host / DMZ
JADE Background Service
Prerequisites
See the Using the tutorials with the JADE YADE Client Command Line Interface article for guidelines to setting up and running the tutorial examples.
Instructions for installing, configuring and using the XML Editor can be found in the XML Editor series of articles.
Download file
The configuration described in this tutorial can be downloaded and then directly opened in the XML Editor using the following link:
- sos-berlin_demolocal_2_local_select_adv.xml
File Transfer Examples
Renaming files
The following example combines two renaming possibilities:
- Renaming files
- Adding a date-stamp to file names
See the JADE YADE Parameter Reference - Rename article for a full list of the renaming possibilities.
Example
The example assumes that five test_*.txt files have previously been downloaded to the local ${USERPROFILE}\jade_demo\a
directory before the example is run. Such a download can be carried out using the simple YADE file transfer operation described in The YADE Client Command Line Interface - Tutorial 1 - Getting Started.
The file name part(s) to be replaced is/are defined in the replacing Replacing parameter with round brackets () and the new part(s) in the replacement Replacement parameter, separated by semi-colons.
...
- test will be replaced with file_ and
- '_' with the current date in the format specified.
Configuration structure
ProtocolFragment
ProtocolFragments are not specified in this configuration as both source and target are on the local file system. . This has been done to allow write permissions at both source and target.
Profile
The meaning of the three elements required to specify renaming - Rename, ReplaceWhat and ReplaceWith elements should be self-explanatory.
Note that
- the Rename operation in this example described here is specified for the transfer source, meaning that the source files will be renamed and that the files copied to the target will not be. It is also possible to specify Rename for the transfer target, in which case the source files will remain unchanged for a copy operation.
- write permissions are required before names of files on a file system can be changed.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile used in this example in XML and in settings.ini
formats.
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
[local_2_local_replace_datestamp] operation = copy source_protocol = local source_replacing = ^(test)(_)[0-9]\.txt source_replacement = file_;[date:yyyy-MM-dd]_ file_spec = ^test_[0-9]\.txt$ source_dir = ${USERPROFILE}\jade_demo\a target_protocol = local target_dir = ${USERPROFILE}\jade_demo\b |
Running the Profile
This profile is called on Windows systems using the following command, depending on the JADE YADE version being used:
Code Block | ||||
---|---|---|---|---|
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_select_adv.xml" -profile="local_2_local_replace_datestamp" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_select_adv.ini" -profile="local_2_local_replace_datestamp" |
On Unix systems the profile is called using one of the following commands, again depending on the YADE version being used:
Code Block | ||||
---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/local_2_local_select_adv.xml" -profile="local_2_local_replace_datestamp" ./jade.sh -settings="${HOME}/jade_demo/local_2_local_select_adv.ini" -profile="local_2_local_replace_datestamp" |
Behavior
- The five test_*.txt files in the the a folder will be copied to the b folder under their original names. The source files will then be renamed as described above.
- If a file is found but not transferred - because for example it has zero bytes and TransferZeroByteFiles is set to false then the file name will remain unchanged.
File cumulation
JADE YADE can cumulate the contents of individual files to a single target file:
- Files matching the file_spec FileSpec regular expression are cumulated together into a new file.
- The CumulativeFileDelete element is used to specify whether an already existing CumulativeFile is deleted and a new, empty, CumulativeFile created before the content of the individual files being transferred is added to this file. (The CumulativeFile is by default opened in append mode.)
- The individual, original files can be deleted if required in a Move operation or separate Remove operation.
- There is not a de-cumulate parameter available at the moment.
Example
The example assumes that five test_*.txt files have previously been downloaded to the local ${USERPROFILE}\jade_demo\a
directory before the example is run.
In the example, the contents of these five files written to a CumulativeFile in the local ${USERPROFILE}\jade_demo\b
directory. The content of each file being transferred is added successively to the CumulativeFile in the order in which the files are received. This order cannot be controlled during the file transfer operation.
...
The text specified in the CumulativeFileSeparator parameter is incorporated in the cumulative file the CumulativeFile between the contents of the individual files.
Configuration structure
ProtocolFragment
ProtocolFragments are not specified in this configuration as both source and target are on the local file system. This has been done to allow write permissions at both source and target.
Profile
The three CumulateFile elements are child elements of the TargetFileOptions element as can be seen in the screenshot below.
Note that
- if the Rename operation is specified for the transfer source, then the source files will be renamed and the files copied to the target will not be. It is also possible to specify Rename for the transfer target, in which case the source files will remain unchanged for a Copy operation.
- write permissions are required before names of files on a file system can be changed.
XML Editor Configuration
The Profile Code
The following code boxes can be opened to show the Profile and ProtocolFragments used in this example in XML and in settings.ini
formats.
Code Block | |||||||
---|---|---|---|---|---|---|---|
| |||||||
<?xml version="1.0" encoding="utf-8"?> <Configurations> <Fragments> <ProtocolFragments /> </Fragments> <Profiles> <Profile profile_id="local_2_local_replace_datestamp"> ... </Profile> <Profile profile_id="local_2_local_cumulate"> <Operation> <Copy> <CopySource> <CopySourceFragmentRef> <LocalSource /> </CopySourceFragmentRef> <SourceFileOptions> [sftp_server_2_local_cumulate] ;; operation = copy file_spec<Selection> <FileSpecSelection> = <FileSpec><![CDATA[^test_.\.txt$ ;; cumulate_files [0-9]\.txt$]]></FileSpec> <Directory><![CDATA[${USERPROFILE}\jade_demo\a]]></Directory> </FileSpecSelection> </Selection> </SourceFileOptions> </CopySource> <CopyTarget> <CopyTargetFragmentRef> <LocalTarget /> = true cumulative_file_name</CopyTargetFragmentRef> <Directory><![CDATA[${USERPROFILE}\jade_demo\b]]></Directory> <TargetFileOptions> = text-files.txt cumulative_file_separator = <CumulateFiles> <CumulativeFileSeparator><![CDATA[--- File: %{SourceFileName} --- ;;cumulative_file_delete = true ;; source_host]]></CumulativeFileSeparator> <CumulativeFilename><![CDATA[text-files.txt]]></CumulativeFilename> <CumulativeFileDelete>true</CumulativeFileDelete> </CumulateFiles> </TargetFileOptions> = test.sos-berlin.com source_protocol </CopyTarget> </Copy> </Operation> </Profile> </Profiles> </Configurations> |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
[local_2_local_replace_datestamp] ... [local_2_local_cumulate] operation = sftp source_ssh_auth_method = password source_port = 22copy source_userprotocol = demo source_passwordlocal file_spec = demo ^test_[0-9]\.txt$ source_dir = / ;; target_host = ${USERPROFILE}\jade_demo\a target_protocol = localhost target_protocol = local target_dir = ${USERPROFILE}\jade_demo\ab cumulate_files = true cumulative_file_separator = --- File: %{SourceFileName} --- cumulative_file_name = text-files.txt cumulative_file_delete = true |
Running the Profile
This profile is called on Windows systems using the following command, depending on the YADE version being usedRun the example using:
Code Block | ||||
---|---|---|---|---|
| ||||
jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_select_adv.xml" -profile="local_2_local_cumulate" jade.cmd -settings="%USERPROFILE%\jade_demo\local_2_local_select_adv.ini" -profile="local_2_local_cumulate" |
Code Block | ||||
---|---|---|---|---|
| ||||
./jade.sh -settings="${HOME}/jade_demo/local_2_local_select_adv.xml" -profile="sftp_serverlocal_2_local_cumulate" ./jade.sh -settings="${HOME}/jade_demo/local_2_local_select_adv.ini" -profile="local_2_local_cumulate" |
Behavior
Opening the cumulative file text-files.txt in a text editor after transfer has been completed shows will show the content of all five individual files.
Note that the order of arrival of files in the target system determines the order in which individual files were they are added to the cumulative file.Note that this will not necessarily follow the order in which transfer of individual files was started
Setting the CumulativeFileDelete parameter to false and running the transfer operation a number of times will show the that this setting allows the ongoing addition of content to the CumulativeFile - a procedure that is commonly used, for example, for log files.