Table of Contents | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
Scenario
In this example user we want to poll for an incoming file on the ServerAServer A, once file(s) matching with the a reguler expression is are found on ServerA, Server A then JobScheduler should transfer the file(s) from ServerA Server A to ServerBServer B.
After successfully successful transfer of file on ServerB, JobScheduler should start an processing script on ServerB files to Server B then JobScheduler should execute a script for post-processing on Server B via SSH.
Graphviz |
---|
digraph "Example: Server to Server Transfer with polling" \{ rankdir=LR; edge [ color="#31CEF0" ]; source_server [label="DataServerA\nTransfer Source (FTP)"]; target_server [label="DataServerB\nTransfer Target (sFTPSFTP)"]; jobscheduler [label="JobScheduler \n JADEYADE JITL job \n SSH JITL Job"]; source_server->target_server [label="2. datafile transfer"]; source_server->jobscheduler [dir=both label="commands" color="grey"]; sourcejobscheduler->source_server->jobscheduler [dir=bothright label="1. polling" color="blue"]; jobscheduler->target_server [dir=both label="commands" color="grey"]; jobscheduler->target_server [dir=righ label="3. ssh command" color="red"]; } |
Solution
- For such a file transfer and processing workflow
...
- JobScheduler has built-in JITL jobs, namely
...
...
- YADE JITL Jobs be configured to transfer data from Server-to-Server without
...
- touchdown, more information about Server-to-Server transfer can be found here: Example for YADE Server-to-Server File Transfer.
...
- YADE has a built-in polling feature
...
- that can be configured by using polling
...
- parameters, i.e.
- poll_interval: frequency of polling in seconds and
- poll_timeout: maximum
...
- duration in seconds for polling.
- The job
...
- chain can be configured by an order to run at a predefined time, to repeat execution or
...
- to be started manually.
Job Chain: FileTransferAndProcessing.job_chain.xml
The job chain FileTransferAndProcessing has two nodes:
- jade_s2s_file_transfer: for server to server data transfer and
- ssh_start_proccessing: starting the processing script using ssh
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="ISO-8859-1"?> <job_chain orders_recoverable="yes" visible="yes" title="Tranasfer Data from ServerA to ServerB and start Processing on ServerB vis SSH" max_orders="1" name="DataTransferProcessing">> <job_chain_node state="jade_s2s_file_transfer" job="jade_s2s_file_transfer" error_state="error" next_state="ssh_start_proccessing" on_error="suspend"/> <job_chain_node state="ssh_start_proccessing" job="ssh_start_proccessing" next_state="success" error_state="error" on_error="suspend"/> <job_chain_node state="success"/> <job_chain_node state="error"/> </job_chain> |
Step 1: Job jade_s2s_file_transfer
...
- This job is
...
- a YADE JITL job
...
- that is configured to poll a source
...
- host (
...
- via FTP) for
...
- files matching file_spec=^TEST*\.log$ for every 30 seconds
...
- during a maximum of 600 seconds.
...
- The matching file(s)
...
- will be
...
- copied to the target
...
- host (
...
- SFTP).
...
- More information about
...
- YADE JITL job configuration can be found here
...
Code Block | |||||||
---|---|---|---|---|---|---|---|
| |||||||
<?xml version="1.0" encoding="ISO-8859-1"?> <job title="S2S transfer with polling (API Job for JobScheduler Advanced Data Exchange)" order="yes" stop_on_error="no" name="jade_s2s_file_transfer"> <description > <include file="jobs/jadeJob.xml"/> </description> <params > <param name="file_spec" value="\.log$"/> <param name="operation" value="copy"/> <param name="verbose" value="9"/> <param name="poll_interval" value="30"/> <param name="poll_minfiles" value="1"/> <param name="poll_timeout" value="600"/> <param name="source_protocol" value="ftp"/> <param name="source_host" value="mp.sos"/> <param name="source_user" value="test"/> <param name="source_password" value="super_secret"/> <param name="source_dir" value="/outbound"/> <param name="target_protocol" value="sftp"/> <param name="target_host" value="homer.sos"/> <param name="target_user" value="test"/> <param name="target_password" value="super_secret"/> <param name="target_ssh_auth_method" value="password"/> <param name="target_dir" value="/home/test/inbound"/> </params> <script language="java" java_class="sos.scheduler.jade.JadeJob"/> <run_time /> </job> |
Step 2: Job ssh_start_proccessing
...
- This job is
...
- a JITL job
...
- that is configured to execute
...
- a command on a remote
...
- host using
...
- SSH. The SSH JITL job can be configured to connect using
...
- a public/
...
- private key pair
...
- or with
...
- user/password authentication.
- More information about SSH JITL job configuration can be found here: SSH_JITL_JOB_PARAMETERS.
Code Block | |||||||
---|---|---|---|---|---|---|---|
| |||||||
<?xml version="1.0" encoding="ISO-8859-1"?> <job title="Launch commands or executable files by SSH" order="yes" name="ssh_start_proccessing"> <description > <include file="jobs/SOSSSHJob2JSAdapter.xml"/> </description> <params > <param name="host" value="homer.sos"/> <param name="port" value="22"/> <param name="user" value="test"/> <param name="auth_method" value="password"/> <param name="command" value="uname -an"/> <param name="password" value="super_secret"/> </params> <script language="java" java_class="sos.scheduler.job.SOSSSHJob2JSAdapter"/> <run_time /> </job> |
Also see
Download example
...
Related Downloads
- JobScheduler job_chain and jobs: download example
See also
- YADE Implementation Architecture
- How to transfer files with YADE from Server A to Server B using polling and post-processing
- Example for YADE Server-to-Server File Transfer
- Example for YADE Server-to-Server File Transfer with polling