Scenario
In this example user want to poll for an incoming file on the ServerA, once file matching with the reguler expression is found on ServerA, JobScheduler should transfer the file(s) from ServerA to ServerB.
After successfully transfer of file on ServerB, JobScheduler should start an processing script on ServerB via SSH.
Solution
For such a file transfer and processing workflow, JobScheduler has built in JITL jobs, namely JADE job and SSH job.
The JADE JITL jobs can be configured to transfer data from Server-to-Server without touch down, more information about Server-to-Server can be found here Server to Server Transfer.
The JADE has inbuilt polling feature which can be configured using polling parameter of JADE i.e. poll_interval : frequency of polling in seconds and poll_timeout : maximum time in seconds for polling. The job_chain can be configured by an order to run at predefined time, repeat execution or can be started manually.
Job Chain : FileTransferAndProcessing.job_chain.xml
The job chain FileTransferAndProcessing has two nodes:
- jade_s2s_file_transfer : for server to server data transfer and
- ssh_start_proccessing : starting the processing script using ssh
<?xml version="1.0" encoding="ISO-8859-1"?> <job_chain orders_recoverable="yes" visible="yes" title="Tranasfer Data from ServerA to ServerB and start Processing on ServerB vis SSH" max_orders="1" name="DataTransferProcessing"> <job_chain_node state="jade_s2s_file_transfer" job="jade_s2s_file_transfer" error_state="error" next_state="ssh_start_proccessing" on_error="suspend"/> <job_chain_node state="ssh_start_proccessing" job="ssh_start_proccessing" next_state="success" error_state="error" on_error="suspend"/> <job_chain_node state="success"/> <job_chain_node state="error"/> </job_chain>
Step 1: jade_s2s_file_transfer.job.xml
This job is an JADE JITL job which is configured to poll source server (ftp) for file matching with file_spec=^TEST*\.log$ for every 30 seconds for maximum of 600 seconds.
All the matching file during polling will be copy to the target server (sftp). For more information about JADE JITL can be found here JADE_JITL_Parameters.
<?xml version="1.0" encoding="ISO-8859-1"?> <job title="S2S transfer with polling (API Job for JobScheduler Advanced Data Exchange)" order="yes" stop_on_error="no" name="jade_s2s_file_transfer"> <description > <include file="jobs/jadeJob.xml"/> </description> <params > <param name="file_spec" value="\.log$"/> <param name="operation" value="copy"/> <param name="verbose" value="9"/> <!-- -> <param name="poll_interval" value="30"/> <param name="poll_minfiles" value="1"/> <param name="poll_timeout" value="600"/> <param name="source_protocol" value="ftp"/> <param name="source_host" value="mp.sos"/> <param name="source_user" value="test"/> <param name="source_password" value="test"/> <param name="source_dir" value="/outbound"/> <param name="target_protocol" value="sftp"/> <param name="target_host" value="homer.sos"/> <param name="target_user" value="test"/> <param name="target_password" value="super_secret"/> <param name="target_ssh_auth_method" value="password"/> <param name="target_dir" value="/home/test/inbound"/> </params> <script language="java" java_class="sos.scheduler.jade.JadeJob"/> <run_time /> </job>
Step 2: ssh_start_proccessing.job.xml
This job is an JITL job which is configured to execute an command on remote server using ssh. The SSH JITL job can be configured to connect using Private/public key pair, or with userid/password.
More information about SSH JITL job can be found here SSH_JITL_JOB_PARAMETERS.
<?xml version="1.0" encoding="ISO-8859-1"?> <job title="Launch commands or executable files by SSH" order="yes" name="ssh_start_proccessing"> <description > <include file="jobs/SOSSSHJob2JSAdapter.xml"/> </description> <params > <param name="host" value="homer.sos"/> <param name="port" value="22"/> <param name="user" value="test"/> <param name="auth_method" value="password"/> <param name="command" value="uname -an"/> <param name="password" value="super_secret"/> </params> <script language="java" java_class="sos.scheduler.job.SOSSSHJob2JSAdapter"/> <run_time /> </job>