Page History
Table of Contents |
---|
Introduction
- JS7 make makes use of the YADE Add-on for transfer of files by use of transferring files using a number of protocols such as FTP, FTPS, SFTP, WebDAV, HTTP/HTTPS etc.
- The JS7 Agent ships with a YADE client installation that is available from in the
yade
sub-directory of the Agent's home directory. - In addition to the JS7 - Order History and JS7 - Task History, the JS7 - File Transfer History is provided for displaying detailed results about transfers and the files included.
- For File Transfers:
- The File Transfer View provides information about past and present transfers.
- Similar information is available from the JS7 - File Transfer History view.
- The File Transfer Configuration is used to manage the configuration, for example to access SFTP servers and to select files and locations for transfer.
- For details see JS7 - File Transfer Configuration and JS7 - File Transfer - Deployment.
- File Transfer Jobs can share a File Transfer configuration and perform the effective transfer.
- The File Transfer View provides information about past and present transfers.
File Transfer View
The File Transfer view allows current transfers to be monitored and allows past and present transfers to be checked:
Explanation:
- The view shows details of each transfer such as:
- the source, target and transfer result. These are reported individually for each file included in a transfer,
- any errors occurring during transfer.
- The Advanced Filter allows individual files to be looked up by name, path etc.
- Find more details from the JS7 - File Transfer History view.
File Transfer Configuration
The Configuration=>File Transfer sub-view offers a schema aware configuration editor for File Transfers like this:
The configuration editor is used, for example, to manage connection data and credentials for access to remote servers. In addition, it allows management of profiles that specify which servers are involved in a transfer and which files should be transferred.
Find details from the JS7 - File Transfer Configuration article.
File Transfer Jobs
Location of the YADE Client
The YADE Client ships with Agents by default from the installation directory:
/opt/sos-berlin.com/js7/agent/yade
(Unix)C:\Program Files\sos-berlin.com\js7\agent\yade
(Windows)
Usage of the YADE Client
File Transfer jobs are technically shell jobs that execute the YADE command line interface (CLI). Therefore the job script looks will look like this:
:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
#!/bin/sh
$JS7_YADE_BIN \
-settings $JS7_YADE_CONFIG_DIR/yade.xml \
-profile transfer_by_sftp \
-file_path /tmp/some.file.txt |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
%JS7_YADE_BIN% ^
-settings %JS7_YADE_CONFIG_DIR%\yade.xml ^
-profile transfer_by_sftp ^
-file_path C:\tmp\some.file.txt |
ExplanationExplanations:
- The job script can make use of a number of JS7 - Job Environment Variables that are automatically available. This includes any environment variables prefixed with
JS7
such as:
JS7_YADE_BIN
, which points to the YADE start script that is available:- for Unix from
$JS7_AGENT_HOME/yade/bin/yade.sh
- for Windows from
%JS7_AGENT_HOME%/\yade/\bin/\yade.cmd
- for Unix from
JS7_YADE_DMZ_BIN
, which points to the second YADE start script that is used for transfers with a YADE Jump Host available:- for Unix from
$JS7_AGENT_HOME/yade/bin/yade4dmz.sh
- for Windows from
%JS7_AGENT_HOME%\yade\bin\yade4dmz.shcmd
- for Unix from
JS7_YADE_CONFIG_DIR
, which points to the YADE configuration directory that corresponds to the Agent's configuration directory, i.e.JS7_AGENT_CONFIG_DIR
.
- The following environment variables are specific to this File Transfer job as they are mapped from order variables and node arguments, see JS7 - Order Variables:arguments are used:
-settings
expects the path to a YADE configuration file (*.xml, *.ini) that specifies the file transfer source, targets and profiles.-profile
expectsYADE_PROFILE
is the name of the profile to be applied from the the YADE configuration file.YADE_FILE_PATH
- is
-file_path
can optionally be used as the path of the a file to be transferred if not otherwise specified in the profile.
Parameterization of the YADE Client
Mapping of Variables
- The mapping of order variables and optionally node arguments to environment variables for the a job looks like this:
- The declaration of the above order variables looks will look like this:
- Consider Note that use of default values is not required. If no default value is specified then the order has to specify a value for the respective variable in question.
...
Command Line Arguments
The YADE Client offers to specify allows a number of additional command line arguments to be specified:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
#!/bin/sh
# Any environment variables assigned the job are automatically applied if used by the YADE profile
$JS7_YADE_BIN \
-settings $JS7_YADE_CONFIG_DIR/yade.xml \
-profile $YADE_PROFILE \
-java-options "-Xmx32m" \
-log-level debug |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
@rem Any environment variables assigned the job are automatically applied if used by the YADE profile
%JS7_YADE_BIN% ^
-settings %JS7_YADE_CONFIG_DIR%\yade.xml ^
-profile %YADE_PROFILE% ^
-java-options "-Xmx32m" ^
-log-level debug |
ExplanationExplanations:
- The following command line arguments can be used with a File Transfer job:
-settings
is the path to a YADE *.xml or *.ini configuration file that holds fragments and profiles for transfer of files.- YADE configurations can be managed from the Configuration -> File Transfer View and are offered for export and can be both automatically deployed and be exported to *.xml files:
- For details about how to deploy YADE configurations see JS7 - File Transfer Deployment.
- As the The YADE configuration file has to be located with an Agent we recommend to manually deploy or transfer , therefore transferring this file to the respective an Agent is an option.
-profile
is the name of the profile to be applied from the the YADE configuration file.-java-options
are options for the Java Virtual Machine. Frequently such options are used to specify the maximum heap size. By default YADE will use 32 MB, however, if 1000s of files are transferred with a single call to the YADE Client, e.g. when using a file specification to transfer any all the files from a directory, then it is required will be necessary to increase the default heap size.-log-level
enables to specify Log4j2 log level indicators such asinfo, debug, trace
that to be specified. These will provide more detailed information, e.g. in case that files permanently could not be transferred.for example in the event of files which permanently cannot be transferred. You will find more details in JS7 - Log Levels and Debug Options article.
- The job script is not not indicating shown above does not indicate a specific selection of files for transfer. This gives you two options:
- Hardto hard-wire the file selection directly with the YADE configuration file (yade.xml).
- Specify to specify the file selection from an order variable that is mapped to an environment variable with the job script. Use this environment variable with the YADE configuration.
- Example:
- Add an environment variable such as e.g.
YADE_FILE_SPEC
to your mapping of environment variables.- The value can be a hard-wired regular expression:
- The value can be mapped from an order variable or node argument:
- The value can be a hard-wired regular expression:
- The YADE configuration can makes use of the
${YADE_FILE_SPEC}
environment variable that will be substituted at run-time, e.g. with: - Consider Note that this mechanism can be used for any environment variable names variables that are available for a job and with the YADE configuration.
- Add an environment variable such as e.g.
- Example:
...
...
File Transfer
...
Explanations:
- The view offers details about each transfer such as
- the source, target, transfer result that are reported individually for each file included with a transfer,
- any errors occurring during transfer.
- The Advanced Search offers to look up individual files by name, path etc.
File Transfer Security
The YADE supports a number of mechanisms to secure file transfer operation:
- Connection Management
- Use of secure protocols such as SFTP, FTPS, WebDAVSWebDAV
- Use of private/public keys for authentication
- Use of a a YADE Credential Store for for file transfer connections
- Access Management
- Access to JOC Cockpit is subject to secure authentication mechanisms including JOC Cockpit - Two-factor AuthenticationAccess to file transfer information in JOC Cockpit is subject to JS7 - Default Roles and Permissions
- Logging
- The JS7 - Logging stores information about access by users, including connection attempts and permission violations.
- The JS7 - Audit Log stores information about any operations on file transfers such as starting, stopping a transfer.
- The JS7 - History holds and JS7 - File Transfer History hold information about past file transfers.
The above building blocks allow to implement the implementation of secure file transfers for compliance with standards such as HIPPA, PCI-DSS, SOX etc. For further information see Is YADE compliant to some information security standard?
File Transfer Operation
Parallelism of File Transfers
...
- Memory requirements typically include 32 MB for a YADE job.
- CPU consumption is somewhat higher when loading the YADE Client, however this will be modest during the transfer of a file.
YADE jobs are executed for in separate OS processes. Therefore there is no interference between jobs.
...
In this situation consider to use the JS7 - Lock InstructionResource Locks to implement an exclusive lock or a shared lock that will limit parallel access to a resource. Any file transfer jobs that cannot immediately acquire a resource lock Resource Lock will wait until the lock is freed.