Introduction
The YADE File Transfer capabilities cover a larger number of use cases. At the same time users might consider using alternative tools as an equivalent solution for certain types of file transfers:
- The
curl
command to download or upload files can be considered a file transfer. - Use of operating system commands such as
cp
andmv
to copy and to move files can be considered a file transfer. - Individual scripts for similar purposes can be considered a file transfer.
While file transfers from commands and from scripts can easily be added to shell jobs they lack support for the JS7 - File Transfer History:
- The history keeps track of transfers and holds metadata such as the point in time of transfer and duration of transfer.
- The history holds the details for each transferred file.
- The history offers the functionality to search for transferred files.
The JS7 offers the File Transfer History Script to support 3rd party utilities in populating the File Transfer History.
FEATURE AVAILABILITY STARTING FROM RELEASE 2.5.2
- JS-2036Getting issue details... STATUS
Users of earlier JS7 releases find a patch for download and instructions how to apply the patch for JS7 Agents from the above link.
File Transfer History Script
The File Transfer History Script is available with JS7 Agents from the JS7_AGENT_HOME/bin
directory.
Usage
Invoking the File Transfer History Script without arguments displays the usage clause:
Usage: file_transfer_history.sh|.cmd [Options] [Switches] Options: --transfer-file=<transfer-spec> | optional: the transfer specification can occur any number of times and is made up of the elements: <source-file>,<target-file>[,<file-size>[,<error-message>]] --operation=<operation-spec> | optional: operation to copy, to move, to remove or to get a list of files: copy|move|remove|getlist, default: copy --source-account=<account> | optional: account used for authentication with the system holding source files --source-protocol=<protocol-spec> | optional: one of the protocols for access to source files: local|ftp|ftps|sftp|ssh|http|https|webdav|webdavs|smb, default: local --source-host=<hostname> | optional: hostname, IP address or URL of the host holding source files, default: localhost --source-port=<number> | optional: port used to connect to the system holding source files, default: 0 --target-account=<account> | optional: account used for authentication with the system holding target files --target-protocol=<protocol-spec> | optional: one of the protocols for access to target files: local|ftp|ftps|sftp|ssh|http|https|webdav|webdavs|smb, default: local --target-host=<hostname> | optional: hostname, IP address or URL of the host holding target files, default: localhost --target-port=<number> | optional: port used to connect to the system holding target files, default: 0 --start-time=<date-time> | optional: ISO date and time for start time of transfer: yyyy-MM-dd hh:mm:ssZ --end-time=<date-time> | optional: ISO date and time for end time of transfer: yyyy-MM-dd hh:mm:ssZ, default: current time --error=<string> | optional: error message indicating a failed file transfer --delimiter=<character> | optional: delimiter character for entries in the transfer specification, default: comma Switches: -h | --help | displays usage --display-args | displays command line arguments --display-result | displays execution result --dry-run | sets --display-args and --display-result and does not create entries for the File Transfer History
Options
--transfer-file
- Specifies transfer details per file. This option is specified repeatedly if multiple files are transferred.
- Details are specified from the following elements separated by comma as a delimiter:
<source-file>
: the full path to the source file<target-file>
: the full path to the target file<file-size>
: optionally the file size is specified in bytes<error-message>
: optionally an error message is specified signaling a failed transfer
- The delimiter between elements can be specified using
--delimiter
. - Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
- Example:
--transfer-file="/source/a.txt,/target/a.txt"
--transfer-file="/source/b.txt,/target/b.txt,1024,Permission denied"
- First occurrence signals successful transfer of the file, the second occurrence adds an error message signaling failure.
--operation
- Specifies the operation applied to the file transfer which is one of:
copy
,move
,remove
,getlist
. - By default the
copy
operation is assumed.
- Specifies the operation applied to the file transfer which is one of:
--source-account
- Specifies the account that is used when authenticating with the system which holds the source files.
--source-protocol
- Specifies the protocol used to access source files which is one of:
local
: access files using the local file systemftp
: access files using the FTP protocolftps
: access file using the FTPS protocolsftp
: access files using the SFTP protocolssh
: access system using the SSH protocolhttp
: access files using the HTTP protocolhttps
: access files using the HTTPS protocolwebdav
: access files using the WebDAV protocolwebdavs
: access files using the WebDAVs protocolsmb
: access files using the SMB (CIFS) protocol
- Specifies the protocol used to access source files which is one of:
--source-host
- Specifies the hostname, IP address or URL of the host holding source files.
- Examples:
localhost
: specifies the hostname127.0.0.1
: specifies the IP address of the hostlocalhost:80
: specifies the hostname and portftp://user@localhost:21
: specifies the protocol, account, hostname and port
--source-port
- Specifies the port used to connect to the system holding source files.
--target-account
- Specifies the account that is used when authenticating with the system which holds the target files.
--target-protocol
- Specifies the protocol used to access target files.
- For possible values see
--source-protocol
.
--target-host
- Specifies the hostname, IP address or URL of the host holding target files.
- For examples see
--source-host
.
--target-port
- Specifies the port used to connect to the system holding target files.
--start-time
- ISO date and time of transfer begin. The patterns
yyyy-MM-dd hh:mm:ssZ
andyyyy-MM-dd'T'hh:mm:ssZ
are accepted. - Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
- Examples:
2023-01-01 12:00:00
: noon on 1st of January 2023, the time zone of the server operating the JS7 Agent will be used.2023-01-01 12:00:00+0100
: noon on 1st of January 2023 in a time zone UTC+12023-01-01T12:00:00+0100
: same as the previous example, the 'T' character is used to separate date and time.
- ISO date and time of transfer begin. The patterns
--end-time
- ISO date and time of transfer end. The patterns
yyyy-MM-dd hh:mm:ssZ
andyyyy-MM-dd'T'hh:mm:ssZ
are accepted. - Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
- By default the current date and time will be used.
- For examples see
--start-time
.
- ISO date and time of transfer end. The patterns
--error
- Specifies an error message that indicates a failed transfer. The JS7 File Transfer History will qualify the transfer being failed.
- Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
- Example
--error="Permission denied"
--delimiter
- Specifies the delimiter character for entries in the transfer specification
- Values to this option should be quoted using double quotes. For Unix single quotes can be used alternatively.
- By default the comma is used as a delimiter.
- Example
--delimiter=";" --transfer-file="/source/a.txt;/target/a.txt,512"
Switches
-h | --help
- Displays usage.
--display-args
- Displays the command line arguments passed to the File Transfer History command line application.
--display-result
- Displays the resulting File Transfer History values based on the the command line arguments.
--dry-run
- Sets
--display-args and --display-result
and does not create entries for the File Transfer History.
- Sets
Exit Codes
- 0 - Signals success.
- 1 - Signals failed execution.
- Possible reasons:
- the environment variable
JS7_RETURN_VALUES
is not set. - the File Transfer History result can't be serialized/written to a temporary file indicated with the
JS7_RETURN_VALUES
environment variable.
- the environment variable
- Possible reasons:
Examples
The following examples illustrate typical use cases.
Simple Transfer
#!/bin/sh cp -p /tmp/source/a/a1.txt /tmp/target/a/ $JS7_AGENT_HOME/bin/file_transfer_history.sh \ --transfer-file="/tmp/source/a/a1.txt,/tmp/target/a/a1.txt" # copy a file and create an entry in the JS7 File Transfer History
Copy single File
#!/bin/sh cp -p /tmp/source/a/a1.txt /tmp/target/a/ $JS7_AGENT_HOME/bin/file_transfer_history.sh \ --source-account="$(id -u -n)" \ --target-account="$(id -u -n)" \ --target-host="$(hostname -a)" \ --target-protocol="local" \ --transfer-file="/tmp/source/a/a1.txt,/tmp/target/a/a1.txt,42" # copy a file and create an entry in the JS7 File Transfer History # optional arguments for accounts used with source and target systems are specified
Copy single File (from variables)
#!/bin/sh source_dir=${DIR}/source/a source_file=${source_dir}/a1.txt target_dir=${DIR}/target/a cp -p "${source_file}" "${target_dir}"/ $JS7_AGENT_HOME/bin/file_transfer_history.sh \ --source-account="$(id -u -n)" \ --target-account="$(id -u -n)" \ --target-host="$(hostname -a)" \ --target-protocol="local" \ --transfer-file="${source_file},${target_dir}/$(basename ${source_file}),$(wc -c "${source_file}" | cut -d' ' -f 1)" # copy a file and create an entry in the JS7 File Transfer History # the example corresponds to the previous one but makes use of environment variables
Copy multiple Files
#!/bin/bash source_files=$(ls /tmp/source/a/*) target_dir=/tmp/target/a args=() args_count=0 for source_file in ${source_files[@]}; do cp -p "${source_file}" "${target_dir}" args[${args_count}]="--transfer-file=${source_file},${target_dir}/$(basename "${source_file}"),$(wc -c "${source_file}" | cut -d' ' -f 1)" args_count=$(($args_count + 1)) done $JS7_AGENT_HOME/bin/file_transfer_history.sh \ --source-account="$(id -u -n)" \ --target-account="$(id -u -n)" \ --target-host="$(hostname -a)" \ --target-protocol="local" \ "${args[@]}" # copy a number of files and create corresponding entries in the JS7 File Transfer History # for each file a --transfer-file argument is used
Use with Workflows
Integration of the File Transfer History Script with workflows is offered for a number of use cases.
Configuration View
The following workflow example can be uploaded to the Configuration view. The example includes a number of jobs which perform file transfers using Unix cp
commands
Download (upload .json): pdTransferHistory.workflow.json
Explanation:
- The first job
create-files
performs what the title suggests. - The second job
transfer-cp-single
performs transfer of a single file which is straightforward. - The second job
transfer-cp-multi
performs transfer of a number of files and introduces more complexity.- For consistency of transfer the list of available files is created and is used for copy operations and for populating the File Transfer History.
- The third job
transfer-cp-multi-error
performs transfer of a number of files with individually failed transfers.- The job implements the scenario that the second file out of three files cannot be copied.
- The example is somewhat more complex and demonstrates the limits of OS utilities when it comes to error handling.
- The amount of shell code required for proper error handling is increasing and exceeds by far the configuration used for a YADE file transfer.
File Transfer View
File transfers performed by the above workflow are displayed in the File Transfer view:
Explanation:
- The first history entry
- is in a failed state as one out of three transfers failed,
- displays individual files with source, target and transfer status.
- The second history entry
- is in a successful state as all transfers succeeded,
- displays individual files with source, target and transfer status.
- Search operations as offered from the Advanced Filter button can be applied that offer to search for individual files.