Job JobSchedulerFTPSend

Description of JobSchedulerFTPSend - Sends files by FTP

This job is used as a standalone solution or triggered by orders to transfer files via FTP.

This job implements the following features:

• Transfer of partially matched files Alternatively to transferring individual files, you could transfer multiple files by use of regular expressions.

• Recursive transfer of files in directories Files from arbitrarily nested subdirectories can be selected for transfer by regular expressions. Subdirectories will be created at the FTP server if required.

• Atomic Transfer Files can be renamed during transfer in order to make them appear atomically in a target directory. This is relevant e.g. should target directories be monitored for incoming files. This feature guarantees that incoming files are completely transferred before they appear in a directory.

• Handling of empty files There are multiple options to handle empty files: Exclude empty files from transfer, include particular empty files, notification by e-mail in case of transfer of empty files.

• Notification by E-Mail Following the transfer (or absence) of incoming files notifications can be sent by e-mail.

• Renaming of files Files can be renamed after transfer by regular expressions.

• Use of alternative access data and credentials Should a connection or the login to an FTP server fail then alternative access data and credentials can be used, e.g. to connect to a backup FTP server in case of failure. Primary and alternative access parameters are handled additively, i.e. only those alternative parameters have to be specified that differ from the primary access parameters.

• Support for UNC paths in source directories. Files can be directly transferred from a remote server system to an FTP server.

• Massive parallel file transfer For simultaneous transfer of multiple files this job can be parameterized to run in multiple instances: For independent jobs a new job is started per file via <start_job> . For order driven jobs implicit orders are created per file for the current job chain and job node via <add_order> . The current job will be setback until all implicit orders for parallel file transfers are terminated. The initial order for multiple parallel file transfers will be considered as successful if all implicit orders have successfully been completed.

• Send files by FTP or SFTP to a "jump host" and forward them by FTP or SFTP to a target host. Different transfer protocols can be used between local host and "jump_host" and between "jump_host" and target host.

• All parameters are specified on the localhost exclusively, this applies in the same way when using a "jump host" as local parameters are dynamically forwarded to the "jump host".

• Security: no configuration files are used on the "jump host" (except for private key files that are used in order to access a target host); no passwords are stored on the "jump host"; FTP proxy functionality is not used.

• Track record of file transfers Optionally a history file keeps track of every transfer that has been effected. History entries are optionally signalled to a Job Scheduler that adds them to a central database.
  

  Parameter used by JobSchedulerFTPSend

  Parameter Additional parameters

  Any additional parameters that are not preceded by keywords represent paths for files that should be transferred

  Parameter banner_footer

  This program logs output to stdout or to a file that has been specified by the
  parameter log_filename. A template can be used in order to
  organize the output that is created. The output is grouped into
  header, file list and footer.

This parameter specifies a template file for footer output.

Templates can use internal variables and parameters as placeholders in the form %{placeholder}. The standard footer template looks like this:

  *************************************************************************
  execution status = %\{status\}
  successful transfers = %\{successful_transfers\}
  failed transfers = %\{failed_transfers\}
  last error = %\{last_error\}
  

Parameter banner_header

This program logs output to stdout or to a file that has been specified by the parameter log_filename. A template can be used in order to organize the output that is created. The output is grouped into header, file list and footer.

This parameter specifies a template file for header output.

Templates can use internal variables and parameters as placeholders in the form %{placeholder}. The standard header template looks like this:

  *************************************************************************
  * *
  * SOSFTP - Managed File Transfer Utility *
  * -------------------------------------- *
  * *
  *************************************************************************
  version = %\{version\}
  date = %\{date\} %\{time\}
  operation = %\{operation\}
  protocol = %\{protocol\}
  file specification = %\{file_spec\}
  file path = %\{file_path\}
  source host = %\{localhost\} (%\{local_host_ip\})
  local directory = %\{local_dir\}
  jump host = %\{jump_host\}
  target host = %\{host\} (%\{host_ip\})
  target directory = %\{remote_dir\}
  pid = %\{current_pid\}
  ppid = %\{ppid\}

Parameter classpath_base

The parameter is used during installation of this program on a remote server with the parameter operation=install. This parameter specifies the path of the Java Runtime Environment (JRE) at the remote server and is used if on the remote server a JRE is not included in the system path.

The path of the specified JRE is added to the start script at the remote server (sosftp.cmd respectively sosftp.sh).

Parameter current_pid

This parameter is used for Unix systems and - as opposed to other parameters - is usually specified in the start script sosftp.sh. The value of the environment variable $$ is assigned, that contains the current process id (PID).

The executable File getParentPid.exe deliver the current process id for Windows systems.

The process id is used when writing an entry to a history file for each transfer (see parameter history).

Parameter ftp_account

Account for authorization at the FTP server.

Parameter ftp_alternative_account

Alternative parameter for the primary parameter ftp_account .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_host

Alternative parameter for the primary parameter ftp_host .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_passive_mode

Alternative parameter for the primary parameter ftp_passive_mode .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_password

Alternative parameter for the primary parameter ftp_password .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_port

Alternative parameter for the primary parameter ftp_port .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_remote_dir

Alternative parameter for the primary parameter ftp_remote_dir .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_transfer_mode

Alternative parameter for the primary parameter ftp_transfer_mode .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_alternative_user

Alternative parameter for the primary parameter ftp_user .

The alternative parameters are used solely should the connection to an FTP server fail, e.g. if the FTP server were not available or invalid credentials were used.

Parameter ftp_append_files

This parameter specifies whether the content of the source files should be appended to the target files if the target files exist.

The parameter ftp_overwrite_files will not be effective if this parameter is specified with the value true .
The default value for this parameter is false.

Parameter ftp_atomic_suffix

This parameter specifies whether target files should created with a suffix as "~", and then be renamed to the target file name after the file transfer has been completed.

The desired suffix is specified as the value of this parameter.

This setting is recommended if target directories are monitored by the Job Scheduler.

Parameter ftp_check_size

This parameter determines whether the original file size and the number of bytes transferred should be compared after a file transfer and whether an error should be raised if they do not match.
The default value for this parameter is true.

Parameter ftp_compress_files

This parameter specifies whether files should be compressed for transfer. A gzip-compatible compression is used, no further software components are required.

This parameter cannot be used with the ftp_append_files parameter.

Every file is compressed to a single archive that is named according to the input file and the extension .gz . (The .gz extension is specified using the ftp_compressed_file_extension parameter.)
The default value for this parameter is false.

Parameter ftp_compressed_file_extension

This parameter specifies the file extension should target file compression be specified using the ftp_compress_files parameter.
The default value for this parameter is .gz.

Parameter ftp_file_notification_bcc

This parameter specifies the recipient of a blind carbon copy notification e-mail , should files have been transferred

Parameter ftp_file_notification_body

This parameter specifies the body of a mail notification, should files have been transferred.

The list of files transferred will be automatically added to the mail body.

Parameter ftp_file_notification_cc

This parameter specifies the recipient of a carbon copy notification mail, should files have been transferred.

Parameter ftp_file_notification_subject

This parameter specifies the subject of a mail notification, should files have been transferred.

Parameter ftp_file_notification_to

This parameter specifies the recipient of a notification mail, should files have been transferred.

Parameter ftp_file_path

This parameter accepts the absolute name and path of file that should be transferred. An absolute path has to be specified.

The file will be stored under its name in the directory at the FTP server that has been specified by the parameter ftp_remote_dir .

The following parameters are ignored should this parameter be used: ftp_file_spec and ftp_local_dir .

Parameter ftp_file_spec

This parameter specifies a regular expression to select files to be transferred from a local directory.
The default value for this parameter is .{*}.

Parameter ftp_file_zero_byte_notification_bcc

This parameter specifies the recipient of a blind carbon copy notification e-mail, should zero byte files have been found.

Parameter ftp_file_zero_byte_notification_body

This parameter specifies the body of a mail notification, should zero byte files have been found.

The list of zero byte files found will be added automatically to the mail body.

Parameter ftp_file_zero_byte_notification_cc

This parameter specifies the recipient of a carbon copy mail notification, should zero byte files have been found.

Parameter ftp_file_zero_byte_notification_subject

This parameter specifies the subject of a notification mail, should zero byte files have been found.

Parameter ftp_file_zero_byte_notification_to

This parameter specifies the recipient of a notification e-mail, should zero byte files have been found.

Parameter ftp_file_zero_byte_transfer

This parameter specifies whether zero byte files should be transferred and processed by subsequent jobs. The following settings are available:

• yes : Files with zero byte size are transferred (default).

• no : Files with zero byte size are transferred, if at least one of the files has more than zero byte size.

• strict : Files with zero byte size are not transferred. An error will be raised if any zero byte file is found.

• relaxed : Files with zero byte size will not be transferred. However, no error will be raised should this result in no files being transferred. Should orders be used then they will be marked as being completed.

Usage of this parameter can be refined by the parameter ftp_force_files : should that parameter show the value false , then an order will be continued in case that no files have been transferred.

Note that the ftp_remove_files parameter has unrestricted validity. Files with zero byte size will be removed regardless of whether or not they have been transferred.
The default value for this parameter is yes.

Parameter ftp_force_files

This parameter specifies whether an error should be raised if no files could be found for transfer.

The number of files to be transferred is determined by the ftp_file_spec parameter and can be restricted by the ftp_overwrite_files should ftp_force_files be specified with the value false .
The default value for this parameter is true.

Parameter ftp_keep_connection

Tries to process several orders with the same connection. The connection parameters need to be the same.
The default value for this parameter is false.

Parameter ftp_local_dir

Local directory from which files should be transferred; the selection of files is specified by the parameter ftp_file_spec .

Besides paths in the local file system UNC path names are supported that could be used to address remote server systems: \\somehost\somedirectory can be used in the same way as //somehost/somedirectory to transfer files from a remote server system directly to an FTP server.

Moreover, you could specify URIs for a file schema as in file:////somehost/somedirectory . Please, consider the required number of slashes. URIs for the file parameter are subject to the following limitations due to constraints of the underlying Java JRE:

• File names and path names must not contain any spaces.

• Authentication by authority strings as in file:////user:password@somehost/somedirectory is not supported.

The default value for this parameter is ..

Parameter ftp_no_nlist

The default value for this parameter is false.

Parameter ftp_overwrite_files

This parameter specifies whether existing files on the remote host should be overwritten.
The default value for this parameter is true.

Parameter ftp_parallel

This parameter specifies whether a file transfer should be executed in parallel. If this parameter is assigned the value true and if multiple files were selected for transfer, e.g. when using a regular expression for matching file names by the parameter ftp_file_spec , then this job will automatically launch one successor job per file by [[Param<start_job>|<start_job>]] . Such successor jobs will be executed in parallel.

When used within a job chain then automatically for each file an implicit order will be created for the same job chain and the same job node via [[Param<add_order>|<add_order>]] . This job has to be configured e.g. with the job attribute [[Param<job tasks"5">|<job tasks"5">]] in order to enable five parallel job instances. Should the number of files for parallel transfers exceed the maximum allowed number of parallel tasks then the additional orders will be enqueued and will be processed as soon as a free task becomes available.

With the parameters ftp_parallel_check_retry and ftp_parallel_check_interval the maximum number of trials and the interval between two trials are set in which the completion of parallel transfers is checked. A job chain will be continued only if all dependent orders for parallel transfers have been completed or if the maximum number of trials to check for completion is exceeded.

The initial order will be classified as successful only if all implicit orders for parallel transfers have been successfully completed, otherwise the initial order will switch to the error state of the current job node.
The default value for this parameter is false.

Parameter ftp_parallel_check_interval

This parameter specifies the interval in hours:minutes:seconds between two trials to check if parallel transfers have been completed that were specified by the parameter ftp_parallel .

The maximum number of trials can be set with the parameter ftp_parallel_check_retry .

From left to right the specification of hours and minutes in the parameter value can be skipped, e.g. 60 for 60 seconds, 02:30 for 2.5 minutes.
The default value for this parameter is 00:00:60.

Parameter ftp_parallel_check_retry

This parameter specifies the maximum number of trials to check the completion of all implicit orders for parallel transfers that have been created due to the parameter ftp_parallel .

With the parameter ftp_parallel_check_interval the interval between two trials to check for completion can be set.
The default value for this parameter is 60.

Parameter ftp_passive_mode

Passive mode is often used with firewalls. Valid values are 0 or 1 .
The default value for this parameter is 0.

Parameter ftp_password

Password for authorization at the FTP server.

Parameter ftp_port

Port by which files should be transferred.
The default value for this parameter is 21.

Parameter ftp_protocol

ftp , sftp or ftps
If sftp is chosen, the ssh_* parameters will be considered.
For FTPS, only implicit FTPS (using a dedicated FTPS port) is supported.
Explicit FTPS (Server uses the same port for FTP and FTPS) is not supported.
The default value for this parameter is ftp.

Parameter ftp_recursive

This parameter specifies if files from subdirectories should be transferred recursively. Recursive processing is specified by one of the values yes or true .

Regular expression matches apply to files from subdirectories as specified by the parameter ftp_file_spec .

Parameter ftp_remote_dir

Directory on the FTP server to which files should be transferred.
The default value for this parameter is ./.
This parameter is mandatory.

Parameter ftp_remove_files

This parameter specifies if the local file should be removed after transfer.
The default value for this parameter is false.

Parameter ftp_result_files

This parameter is generated after the file transfer. It contains the number of transferred files.
The default value for this parameter is 0.

Parameter ftp_result_zero_byte_files

This parameter is generated after the file transfer. It contains the number of 0-byte files which were transferred.

Parameter ftp_simple_transfer

The default value for this parameter is false.

Parameter ftp_transfer_mode

Tranfer mode can be either ascii or binary .
The default value for this parameter is binary.

Parameter ftp_user

User name for authorization at the FTP server.
The default value for this parameter is anonymous.
This parameter is mandatory.

Parameter history

This parameter causes a history file to be written in CSV format. The path and name of the history file is specified as value for this parameter. A history record is created for each file that has been transferred.

A history file contains the following columns:

• guid A unique identifier for the history entry. This identifier is used for checking of duplicate entries in combination with Job Scheduler for Managed File Transfer.

• mandator A character that denominates the mandator of a file transfer, see respective parameter.

• transfer_timestamp The point in time when the transfer took place.

• pid The process id of the current process that executes the file transfer, see parameter current_pid.

• ppid The process id of the parent of the process that executes the file transfer, see respective parameter.

• operation One of the operations send or receive, see respective parameter.

• localhost The name of the host on which this program is executed.

• localhost_ip The IP address of the host on which this program is executed.

• local_user The name of the user account for which this program is executed.

• remote_host The name of the host to/from which a transfer is executed, see parameter host.

• remote_host_ip The IP address of the host to/from which a transfer is executed, see parameter host.

• remote_user The name of the user account for the remote host, see parameter user.

• protocol The protocol can be either ftp, sftp or ftps, see respective parameter.

• port The port on the remote host, see respective parameter.

• local_dir The local directory to/from which a file has been transferred, see respective parameter.

• remote_dir The remote directory to/from which a file has been transferred, see respective parameter.

• local_filename For send operations this is the original file name on the local host. For receive operations this is the resulting file name on the local host optionally having applied replacement operations, see parameter replacing.

• remote_filename For send operations this is the resulting file name on the remote host optionally having applied replacement operations, see parameter replacing. For receive operations this is the original file name on the remote host.

• file_size The size of the transferred file in bytes.

• md5 The value of the MD5 hash that is created from the file that was transferred.

• status The status can be either success or error.

• last_error_message Should an error have occurred then the last error message will be given in this column.

• log_filename The name of the log file, see respective parameter.

Parameter history_repeat

The parameter is used in order to synchronize parallel write access to the history file by multiple instances of this program.

This parameter specifies the maximum number of repeat intervals when trying to write to the history file if the history file is locked due to parallel instances of this program.

Parameter history_repeat_interval

The parameter is used in order to synchronize parallel write access to the history file by multiple instances of this program.

This parameter specifies the the interval in seconds of repeated trials to write to the history file if the history file is locked due to parallel instances of this program.

Parameter http_proxy_host

The value of this parameter is the host name or the IP address of a proxy used to create the connection to the SSL server. The use of a proxy is optional.

Parameter jump_command

This parameter specifies a command that is to be executed on the SSH server. Multiple commands can be separated by the command delimiter that is specified using the jump_command_delimiter parameter.

Parameter jump_command_delimiter

Command delimiter characters are specified using this parameter. These delimiters can then be used in the jump_command parameter to seperate multiple commands. These commands are then excecuted in separate SSH sessions.

Parameter jump_command_script

This parameter can be used as an alternative to jump_command, jump_command_delimiter and jump_command_script_file. It contains script code which will be transferred to the remote host as a file and will then be executed there.

Parameter jump_command_script_file

This parameter can be used as an alternative to jump_command, jump_command_delimiter and jump_command_script. It contains the name of a script file, which will be transferred to the remote host and will then be executed there.

Parameter jump_host

When using a jump_host then files are first transferred to this host and then to the target system. Different protocols (FTP/SFTP) can be used for these transfer operations.

Host or IP address of the jump_host from which or to which files should be transferred in a first operation.

Parameter jump_ignore_error

Should the value true be specified, then execution errors caused by commands on the SSH server are ignored. Otherwise such execution errors will be reported.

Parameter jump_ignore_signal

Parameter jump_ignore_stderr

This job checks if any output to stderr has been created by a command that is being executed on the SSH server and reports this as an error.

Should the value true be specified for this parameter, then output to stderr will not be reported as an error by the Job Scheduler

Parameter jump_password

Password for authentication with the jump_host.

Parameter jump_port

Port on the jump_host by which files should be transferred. For FTP this is usually port 21, for SFTP this is usually port 22.

Parameter jump_proxy_host

The value of this parameter is the host name or the IP address of a proxy used in order to establish a connection to the jump host. The use of a proxy is optional.

Parameter jump_proxy_password

This parameter specifies the password for the proxy server user account, should a proxy be used in order to connect to the jump host, see parameter jump_proxy_host.

Parameter jump_proxy_port

This parameter specifies the port of a proxy that is used in order to establish a connection to the jump host, see parameter jump_proxy_host.

Parameter jump_proxy_user

The value of this parameter specifies the user account for authentication by the proxy server should a proxy be used in order to connect to the jump host, see parameter jump_proxy_host.

Parameter jump_ssh_auth_file

This parameter specifies the path and name of a user's private key file used for login to the SSH server of the jump_host.

This parameter must be specified if the publickey authentication method has been specified in the jump_ssh_auth_method parameter.

Should the private key file be secured by a password, then this password has to be specified using the jump_password parameter.

Parameter jump_ssh_auth_method

This parameter specifies the authentication method for the SSH server - the publickey and password methods are supported.

When the publickey authentication method is used, then the path name of the private key file must be set in the jump_ssh_auth_file parameter. Should the private key file be secured by a passphrase then this passphrase has to be specified by the jump_password parameter.

For the password authentication method the password for the user account has to be specified using the jump_password parameter.

The authentication methods that are enabled depend on the SSH server configuration. Not all SSH servers are configured for password authentication.

Parameter jump_user

User name for authentication with the jump_host.

Parameter mandator

This parameter specifies the mandator for which a file transfer is effected. The mandator is added to an optional history file, see parameter history and has no technical relevance for the transfer.

Parameter ppid

This parameter is used for Unix systems and - as opposed to other parameters - is usually specified in the start script sosftp.sh. The value of the environment variable $PPID is assigned, that contains the process id of the current parent process (PPID).

The parent process id is used when writing an entry to a history file for each transfer (see parameter history).

Parameter replacement

String for replacement of the expression replacing .
If multiple groups shall be replaced there must be specified one replacement string per group. These strings are separated by a semicolon ";":
replacement: aa;filename;bb

Supports masks for substitution in the file name with format strings that are enclosed with [and] . The following format strings are supported:

• date format date format must be a valid Java data format string, e.g. yyyyMMddHHmmss , yyyy-MM-dd.HHmmss etc.

• filename Will be substituted by the original file name.

Requires the parameter replacing being specified.

Parameter replacing

Regular Expression for filename replacement with replacement .
If the expression matches the filename then the groups found are replaced.

a)
For replacement "capturing groups" are used. Only the content of the capturing groups is replaced.
Replacements are separated by a semicolon ";". Example:

 [[Param replacing|replacing]] : (1)abc(12)def(.*)
 [[Param replacement|replacement]] : A;BB;CCC

Input file: 1abc12def123.txt
Output file: AabcBBdefCCC

b)
If no "capturing groups" are specified then the entire match is replaced. Example:

 [[Param replacing|replacing]] : Hello
 [[Param replacement|replacement]] : 1234

Input file: Hello_World.txt
Output file: 1234_World.txt

Requires the parameter replacement being specified.

Parameter root

The parameter specifies the directory in which this program is allowed to create temporary files. Temporary files are required if due to the parameter setting jump_host files have to be stored on an intermediary server and will be removed after completion of the transfer.

Without this parameter the temporary directory is used as provided by the operating system.

Parameter scheduler_host

This parameter specifies the host name or IP address of a server for which Job Scheduler is operated for Managed File Transfer.

The contents of an optional history file (see parameter history), is added to a central database by Job Scheduler.

This parameter causes the transfer of the history entries for the current transfer by UDP to Job Scheduler. Should Job Scheduler not be accessible then no errors are reported, instead, the contents of the history will automaticall be processed later on.

Parameter scheduler_job_chain

The name of a job chain for Managed File Transfer with Job Scheduler, see parameter scheduler_host. The job chain accepts history entries and performs an import into a central database.

Parameter scheduler_port

Der Port, für den ein Job Scheduler für Managed File Transfer betrieben wird, siehe Parameter scheduler_host.

Parameter ssh_auth_file

This parameter specifies the path and name of a user's private key file used for registeration on an SSH server.

This parameter must be specified if the publickey authorization method has been specified in the auth_method parameter.

Should the private key file be secured by a password, then this must be specified using the password parameter.

Parameter ssh_auth_method

This parameter specifies the authorization method for the SSH server - the publickey and password methods are supported.

When the publickey authorization method used, then the path name of the private key file must be set in the ssh_auth_file parameter. Should the private key file be secured by a password then this must be specified with the password parameter.

For the password authorization method the password for each user account must be specified using the password parameter.

The authorization methods which are enabled depend on the SSH server configuration. Not all SSH server configurations support the password authorization method.
The default value for this parameter is publickey.

Parameter ssh_proxy_host

The value of this parameter is the host name or the IP address of a proxy used to create the connection to the SSH server. The use of a proxy is optional.

Parameter ssh_proxy_password

This parameter specifies the password for the proxy server user account, should a proxy be used to connect to the SSH server.

Parameter ssh_proxy_port

This parameter specifies the port number of the proxy, should a proxy be used to create the connection to the SSH server.

Parameter ssh_proxy_user

The value of this parameter specifies the user account for authorization by the proxy server should a proxy be used to connect to the SSH server.

Parameter testmode

If this Parameter is set to true then only FTP Connections will be performed.

Parameter transactional

This parameter specifies if file transfers should be operated within a single transaction, i.e. either all files are successfully transferred or none. Should an error occur during a transfer operation then all transfers will be rolled back.

When specifying the value true then the following applies:

• The parameter atomic_suffix has to be specified that causes target files to be created with a suffix such as "~" and that causes the respective files to be renamed to their target file name after the transfer of all files has been successfully completed. If at least one file out of a set of files cannot be transferred successfully then no files will be renamed, instead the temporarily created files are removed from the target system.

• The parameter remove_files that causes files to be removed after successful transfer will be effective only after all files have been successfully transferred. Otherwise no files will be removed.

The default value for this parameter is false.