JADE Parameter Reference - SourceFileOptions

• Schema: Selection, Selection

A file Selection includes specifying one of

• FilePathSelection: an absolute path to a file or a relative path used with a Directory
• FileSpecSelection: a regular expression specifying a pattern for file names that are looked up in a Directory
• FileListSelection: a list of files from a Directory

This parameter specifies the Selection of files from a source by the following settings:

• FilePath: one or a number of file names that can be specified with an absolute or relative path.
• Directory: a folder to which the file selection is applied if no absolute path has been used for the FilePath parameter.

The specified files are expected to be available from the source for transfer by YADE.

• Schema: FilePath, FilePath
• Mapping:
  • file_path
  • FileNames
  • file_names

This parameter is used to specify an individual file or a number of files for processing. It is an alternative to the FileSpecSelection and FileListSelection.

More than one file can be specified by using a ";" between the file names. All files will be processed in the order in which they are defined in this parameter.

This parameter accepts absolute, relative and runtime paths of files that are to be processed.

In case of a relative path:

• if the parameter Directory is specified, the path of each specified file is set relative to that directory.
• otherwise the path is relative to the server's working directory, where the working directory depends on the server configuration.

In case of a runtime path (file to process for a file-order in a file_order_source job chain):

• the value must be set to ${scheduler_file_path}

• Schema: Directory, Directory
• Mapping:
  • dir

This parameter is used to specify a directory on one of the involved servers. Directories can be adressed using a folder structure, e.g. /folder1/folder2.

For Windows systems when used with the LocalSource parameter then a Windows path including a drive letter can be specified. For all server systems the standard syntax using can be used which includes forward slashes to separate folder names.

The directory can be set absolute or relative to the working directory, where the working directory depends on the server configuration. If for example a user "test" connects to an SFTP server then this user might have /home/test/ as a working directory. This applies to servers that use the home directory as the working directory. In this case you can use relative and absolute adressing as in

• data
• /home/test/data

• Schema: Recursive
• Mapping:
  • recursive
  • IncludeSubdirectories
  • include_sub_directories
  • RecurseSubFolders

This parameter specifies wether files from subdirectories should be included recursively. Recursion is effective exclusively if files are found in a directory.

• FileSpecSelection
  • Regular expression matches apply to files from subdirectories as specified by the FileSpec and ExcludedDirectories parameters.
• FilePathSelection/FileListSelection
  • YADE-600. Availability starting with 1.13.18
  • Specifies how the files should be transferred: flat in one Target folder or by creating the same sub-directories in the target system as from the source system.

This parameter specifies the Selection of files from a source by the following settings:

• FileSpec: a regular expression to select files.
• Directory: a folder to which the file selection is applied.

All files with names that correspond to the regular expression are added to a results list and are transferred by YADE.

• Schema: FileSpec, FileSpec
• Mapping:
  • file_spec
  • FileNameRegExp
  • FileNamePatternRegExp

This parameter specifies a regular expression, which is used to select files from a Directory .

All files with names that correspond to the regular expression are added to a results list and are transferred by YADE.

This parameter cannot be used with the HTTPFragment or HTTPSFragment in any file transfer operation as the respective protocols would not support to select files based on wildcards.

• Schema: Directory, Directory
• Mapping:
  • dir

This parameter is used to specify a directory on one of the involved servers. Directories can be adressed using a folder structure, e.g. /folder1/folder2.

For Windows systems when used with the LocalSource parameter then a Windows path including a drive letter can be specified. For all server systems the standard syntax using can be used which includes forward slashes to separate folder names.

The directory can be set absolute or relative to the working directory, where the working directory depends on the server configuration. If for example a user "test" connects to an SFTP server then this user might have /home/test/ as a working directory. This applies to servers that use the home directory as the working directory. In this case you can use relative and absolute adressing as in

• data
• /home/test/data

• Schema: ExcludedDirectories, ExcludedDirectories
• Mapping:
  • excluded_directories
• Release:
  • YADE-601 Availability starting with 1.13.18.

This parameter specifies a regular expression that matches the path to directories to be excluded when recursively traversing a Source directory.

• Schema: Recursive
• Mapping:
  • recursive
  • IncludeSubdirectories
  • include_sub_directories
  • RecurseSubFolders

This parameter specifies wether files from subdirectories should be included recursively. Recursion is effective exclusively if files are found in a directory.

• FileSpecSelection
  • Regular expression matches apply to files from subdirectories as specified by the FileSpec and ExcludedDirectories parameters.
• FilePathSelection/FileListSelection
  • YADE-600. Availability starting with 1.13.18
  • Specifies how the files should be transferred: flat in one Target folder or by creating the same sub-directories in the target system as from the source system.

This parameter specifies the Selection of files from a source by the following settings:

• FileList: specifies a file that contains records with file names in each line. This file is expected to be in reach of the YADE Cllient.
• Directory: a folder to which the file selection is applied.

The specified files are expected to be available from the source for transfer by YADE.

• Schema: FileList, FileList
• Mapping:
  • File_List_Name
  • FileListName

This parameter is used to specify a number of files for processing. It is an alternative to the FileSpecSelection and FilePathSelection.

This parameter specifies a file that contains records with file names in each line. The files can be specified with absolute or relative paths. Each record in this file contains the name of a file which is to be processed. All files in the file list will be processed in the sequence they appear in the file. If a file in the list does not exist then the processing will be aborted.

Future behaviour:
In case of a relative path...

• if the parameter Directory is specified, the path of each specified file is relative to that directory.
• otherwise the path is relative to the servers working directory, where the working directory depends on the server configuration.

• Schema: Directory, Directory
• Mapping:
  • dir

This parameter is used to specify a directory on one of the involved servers. Directories can be adressed using a folder structure, e.g. /folder1/folder2.

For Windows systems when used with the LocalSource parameter then a Windows path including a drive letter can be specified. For all server systems the standard syntax using can be used which includes forward slashes to separate folder names.

The directory can be set absolute or relative to the working directory, where the working directory depends on the server configuration. If for example a user "test" connects to an SFTP server then this user might have /home/test/ as a working directory. This applies to servers that use the home directory as the working directory. In this case you can use relative and absolute adressing as in

• data
• /home/test/data

• Schema: Recursive
• Mapping:
  • recursive
  • IncludeSubdirectories
  • include_sub_directories
  • RecurseSubFolders

This parameter specifies wether files from subdirectories should be included recursively. Recursion is effective exclusively if files are found in a directory.

• FileSpecSelection
  • Regular expression matches apply to files from subdirectories as specified by the FileSpec and ExcludedDirectories parameters.
• FilePathSelection/FileListSelection
  • YADE-600. Availability starting with 1.13.18
  • Specifies how the files should be transferred: flat in one Target folder or by creating the same sub-directories in the target system as from the source system.

• Schema: CheckSteadyState, CheckSteadyState
• Mapping:
  • check_steady_state_of_files

In some file transfer scenarios the recipient of a file does not know when the sender creates the file. In case of (very) large files the recipient may try to read the file before the sender has finished writing it. This can result in the recipient retrieving an incomplete file.

Including the CheckSteadyState parameter ensures that the file is checked at its reception point for completeness before starting the transfer.

Including the CheckSteadyState parameter ensures that the file is checked at its reception point for completeness before starting the transfer.

If the check of the steady state of one incoming file did not succeed successfully then no file will be transferred.

Note that this is not a very reliable approach as the recipient checks the date of last modification and the size of the file. If neither changes during a specified a period of time then the file is assumed to be complete. However, if transmission is terminated without the file being completely written, the network goes down or the processing speed of the file is slow, then the receiver will get a corrupted file.

The shorter the check interval is chosen the more likely it may happen that a file is detected as steady, although this was not yet finished writing.

• Schema: CheckSteadyStateInterval
• Mapping:
  • check_steady_state_interval

The interval in seconds for checking the steady state of incoming files.

• Schema: CheckSteadyStateCount
• Mapping:
  • check_steady_count
  • CheckSteadyCount

The number of tries to check the steady state of incoming files.

• Schema: CheckSteadyStateErrorState
• Mapping:
  • Steady_state_error_state
  • SteadyStateErrorState
  • SteadyErrorState

The next state in a job chain that is used if the check of the steady state of incoming files did not succeed successfully.

This parameter is only available if YADE runs as a JobScheduler job.

• Schema: Directives, Directives

Directives are options that specify the behavior with specific file characteristics:

• DisableErrorOnNoFilesFound: do not raise an error if no files are found on the source server
• TransferZeroByteFiles: specify whether zero byte files should be transferred and processed by subsequent commands

• Schema: DisableErrorOnNoFilesFound, DisableErrorOnNoFilesFound
• Mapping:
  • force_files
  • ErrorOnNoDataFound
  • error_on_no_data_found
  • error_when_no_data_found

This parameter specifies whether an error should be raised if no files can be found for transfer.

• By default an error will be reported if no files are found on the source system which corresponds to the value "false".
• No errors will be reported if no files are found and this parameter is set to "true".

The number of files to transfer is determined by the FileSpec, FilePath or FileList parameters.

This parameter does not affect the DisableOverwriteFiles parameter if no files are found that can be overwritten.

• Schema: TransferZeroByteFiles, TransferZeroByteFiles
• Mapping:
  • zero_byte_transfer
  • TransferZeroByteFiles
  • transfer_zero_byte_files

This parameter specifies whether zero byte files should be transferred and processed by subsequent commands.

The following settings are available:

• "true": files with zero byte size are transferred (default).
• "false": files with zero byte size are transferred if at least one of the selected files is more than zero byte in size.
• "strict": files with zero byte size will not be transferred. An error will be raised if any zero byte file is found.
• "relaxed": files with zero byte size will not be transferred. However, no error will be raised if this results in no files being transferred.

Notes

• Note that the Remove operation has unrestricted validity.
  • Files with zero byte size will be removed regardless of whether or not they have been transferred.

• Schema: Polling, Polling

Polling is the YADE capability to wait for incoming files.

• If a file is found then polling is completed and the file transfer starts.
• If no file is found then YADE would wait for a configurable PollInterval and then repeat the check for incoming files.
• Having exceeded the PollTimeout YADE will not repeat checks for incoming files but will fail with the respective exception.

• Schema: PollInterval
• Mapping:
  • poll_interval

This parameter specifies the time in seconds that will be waited before a new poll for files is carried out.

• Schema: PollTimeout
• Mapping:
  • poll_timeout

This parameter specifies the total length of time in minutes that a file will be polled for.

• If the value of this parameter is zero then polling will not be carried out.
• If a file becomes available within the time specified then it will be transferred, otherwise an error will be raised.
• Polling will be stopped once one (or more) file(s) have been detected.

• Schema: MinFiles
• Mapping:
  • poll_minfiles

This parameter specifies the minimum number of files that have to be found during the polling period in order to cause the transfer to start. This parameter becomes effective with the PollTimeout parameter.

• Schema: WaitForSourceFolder
• Mapping:
  • polling_wait_4_Source_Folder
  • pollingWait4SourceFolder

This parameter determines the behavior of the polling when the source folder doesn't exist. If the value is "true" and the source folder doesn't exist then the polling continues otherwise it raise an error.

• Schema: PollErrorState
• Mapping:
  • Poll_Error_State
  • PollErrorState
  • No_files_state

When YADE runs as a job with JobScheduler and is used for polling then this parameter specifies the node of the current job chain that should be executed after the current YADE job if the timeout as specified with the PollTimeout parameter is exceeded without finding a file.

If no state is specified then an error will be raised if no files are found after timeout.

It is required that the parameter PollingServer is "false".

• Schema: PollingServer
• Mapping:
  • polling_server
  • PollingServer

This parameter specifies YADE to act as a polling server.

• If this value "false" then the polling server stops after the first polling trial. The first polling trial ends if either the PollTimeout is reached or some files (at least MinFiles) are found.
• If this value "true" then the polling server starts polling trials endlessly (see PollForever) or until PollingServerDuration is reached. If some files are found inside a polling trial then these files are transferred before the next polling trial is started.

This setting should be "false" if YADE is executed by JobScheduler as JobScheduler itself has mechanisms to start jobs repeatedly.

• Schema: PollingServerDuration
• Mapping:
  • polling_server_duration
  • pollingServerDuration

This parameter specifies how long the YADE polling server should run.

It is required that the parameter PollingServer is "true".

Supported formats:

• s(n) - seconds, e.g. 30,180...
• m(n):s(n) - minutes:seconds, e.g. 10:30, 123:456 ...
• h(n):m(n):s(n) - hours:minutes:seconds, e.g. 12:30:30, 123456:7:8 ...

• Schema: PollForever
• Mapping:
  • polling_server_poll_forever
  • PollingServerPollForever

This parameter specifies an unbounded duration for Polling and would cause YADE not to terminate after processing of incoming files.

This parameter is an alternative to the PollingServerDuration that specifies a limited duration.

It is required that the parameter PollingServer is "true".

This setting should not be used if YADE is executed by JobScheduler as JobScheduler itself has mechanisms to start jobs repeatedly.

• Schema: ResultSet, ResultSet
• Mapping:
  • create_result_set
  • CreateResultSet
  • CreateResultList
  • create_result_list

This handles the result set of a Selection of files from a source host. The result set can be written into a file or checked against an expectation.

• Schema: ResultSetFile
• Mapping:
  • result_set_file_name

If the value of this parameter specifies a valid filename then the result set of a Selection of files from a source host will be written to this file.

The result set file is located on the host that runs the YADE Client.

Check number of expected hits in result-list.

• Schema: ExpectedResultSetCount
• Mapping:
  • expected_size_of_result_set

Number of expected hits in result-list.

• Schema: RaiseErrorIfResultSetIs
• Mapping:
  • raise_error_if_result_set_is

This parameter has only an effect if the YADE Client is operated with JobScheduler.

This parameter specified a comparison operator. With this parameter YADE can be caused to raise an error if the number of files that are subject to a file transfer operation does not correspond to the expected number of files.

Use of this parameter requires

• the number of expected files to be specified with the ExpectedSizeOfResultSet parameter

At run-time the specified value is compared to the effective number of files. In case of a missing match an error is raised.

Possible values of this parameter are:

• for equal
  • =
  • ==
  • eq
  • equal
• for not equal
  • !=
  • <>
  • ne
  • not equal
• for less than
  • <
  • lt
  • less than
• for less or equal
  • <=
  • le
  • less or equal
• for greater or equal
  • >=
  • ge
  • greater or equal
• for greater than
  • >
  • gt
  • greater than

Example:

• Assuming that the parameter RaiseErrorIfResultSetIs is set "=" and the parameter ExpectedSizeOfResultSet is set to the value "1".
• If the number of files does not match "1" then an error will raised.

• Schema: EmptyResultSetState
• Mapping:
  • on_empty_result_set

This parameter is used to specify the processing if

• no files are found that are subject to a file transfer operation and
• the YADE Client is operated in job chain with JobScheduler.

The respective job chain node is expected to handle the situation of missing files, e.g. to repeat processing.

The respective job chain node is only reached if no errors occurred. In particular, please note that the parameter DisableErrorOnNoFilesFound has to be "true".

• Schema: MaxFiles
• Mapping:
  • Max_Files
  • MaxFiles

This parameter specifies the maximum number of files that are considered for transfer. If additional files were available then they will be ignored.

Note that it is not possible to specify which files will be ignored should the value set for this parameter be exceeded. It therefore only makes sense to use this parameter in particular situations such as when the contents of a directory will be repeatedly polled.

• Schema: CheckIntegrityHash, CheckIntegrityHash
• Mapping:
  • check_security_hash

This parameter makes use of a checksum file that is available with the source of a file transfer. The name of the checksum file is assumed to be the same as the source file with an additional extension .md5. During file transfer the contents of the checksum file is compared with the checksum that is calculated from the transfer of the file.

When used with a jump host then integrity checking applies to source and jump host, not to the target of the transfer.

With this element being used a checksum file is expected on the source system and the integrity hash for the target file is calculated and compared with the respective integrity hash of the source file. If the hashes are not equal then the file transfer will be rolled back. If the checksum file on the source system is missing then this will be reported as an information but will not affect the transfer of files.

• Schema: HashAlgorithm, HashAlgorithm
• Mapping:
  • security_hash_type
  • SecurityHashType

This parameter specifies the algorithm that is used to create the checksum. At the time of writing md5 only is supported. If the parameters CheckIntegrityHash or CreateIntegrityHashFile are used then the integrity hash is calculated and compared according to this parameter.