Table of Contents |
---|
| outlinh1. true |
---|
| outlinh1. true |
---|
1 | printablefalse |
---|
2 | stylh1. none |
---|
3 | indent20px |
---|
|
...
Scenario
In this example user we want to poll for an incoming file on the ServerAServer A, once file(s) matching with a reguler expression is are found on Server A then JobScheduler should transfer the file(s) from ServerA Server A to ServerBServer B.
After successfully successful transfer of
file on ServerB, JobScheduler should start an processing script on ServerB files to Server B then JobScheduler should execute a script for post-processing on Server B via SSH.
Graphviz |
---|
digraph "Example: Server to Server Transfer with polling" {
rankdir=LR;
edge [color="#31CEF0"];
source_server [label="ServerA\nTransfer Source (FTP)"];
target_server [label="ServerB\nTransfer Target (SFTP)"];
jobscheduler [label="JobScheduler \n YADE JITL job \n SSH JITL Job"];
source_server->target_server [label="2. file transfer"];
source_server->jobscheduler [dir=both label="commands" color="grey"];
jobscheduler->source_server [dir=right label="1. polling" color="blue"];
jobscheduler->target_server [dir=both label="commands" color="grey"];
jobscheduler->target_server [dir=righ label="3. ssh command" color="red"];
} |
Solution
- For such a file transfer and processing workflow
...
- JobScheduler has built-in JITL jobs,
...
- namely the YADE job and SSH job.
- The YADE JITL Jobs be configured to transfer data from Server-to-Server without touchdown, more information about Server-to-Server transfer can be found here: Example for YADE Server-to-Server File Transfer.
- YADE has a built-in polling feature that can be configured by using polling parameters, i.e.
- poll_interval: frequency of polling in seconds and
- poll_timeout: maximum duration in seconds for polling.
- The job chain can be configured by an order to run at a predefined time, to repeat execution or to be started manually.
Job Chain: FileTransferAndProcessing.job_chain.xml
The job chain FileTransferAndProcessing will have two nodes/steps
Step 1: jade_s2sFileTransfer.job.xml
Step 2: start_proccessing_ssh.job.xml
Jade Job : JADE_S2SFileTransfer.job.xml
has two nodes:
- jade_s2s_file_transfer: for server to server data transfer and
- ssh_start_proccessing: starting the processing script using ssh
Code Block |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job_chain orders_recoverable="yes" visible="yes" title="Tranasfer Data from ServerA to ServerB and start Processing on ServerB vis SSH">
<job_chain_node state="jade_s2s_file_transfer" job="jade_s2s_file_transfer" error_state="error" next_state="ssh_start_proccessing" on_error="suspend"/>
<job_chain_node state="ssh_start_proccessing" job="ssh_start_proccessing" next_state="success" error_state="error" on_error="suspend"/>
<job_chain_node state="success"/>
<job_chain_node state="error"/>
</job_chain>
|
Step 1: Job jade_s2s_file_transfer
- This job is a YADE JITL job that is configured to poll a source host (via FTP) for files matching file_spec=^TEST*\.log$ for every 30 seconds during a maximum of 600 seconds.
- The matching file(s) will be copied to the target host (SFTP). More information about YADE JITL job configuration can be found here: JITL Jobs by topic.
Code Block |
---|
language | xml |
---|
title | Job jade_s2s_file_transfer |
---|
collapse | true |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job title="S2S transfer with polling (API Job for JobScheduler Advanced Data Exchange)" order="yes">
<description >
<include file="jobs/jadeJob.xml"/>
</description>
<params >
<param name="file_spec" value="\.log$"/>
<param name="operation" value="copy"/>
<param name="verbose" value="9"/>
<param name="poll_interval" value="30"/>
<param name="poll_minfiles" value="1"/>
<param name="poll_timeout" value="600"/>
<param name="source_protocol" value="ftp"/>
<param name="source_host" value="mp.sos"/>
<param name="source_user" value="test"/>
<param name="source_password" value="super_secret"/>
<param name="source_dir" value="/outbound"/>
<param name="target_protocol" value="sftp"/>
<param name="target_host" value="homer.sos"/>
<param name="target_user" value="test"/>
<param name="target_password" value="super_secret"/>
<param name="target_ssh_auth_method" value="password"/>
<param name="target_dir" value="/home/test/inbound"/>
</params>
<script language="java" java_class="sos.scheduler.jade.JadeJob"/>
<run_time />
</job>
|
Step 2: Job ssh_start_proccessing
- This job is a JITL job that is configured to execute a command on a remote host using SSH. The SSH JITL job can be configured to connect using a public/private key pair or with user/password authentication.
- More information about SSH JITL job configuration can be found here: SSH_JITL_JOB_PARAMETERS.
Code Block |
---|
language | xml |
---|
title | Job ssh_start_proccessing |
---|
collapse | true |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job title="Launch commands or executable files by SSH" order="yes">
<description >
<include file="jobs/SOSSSHJob2JSAdapter.xml"/>
</description>
<params >
<param name="host" value="homer.sos"/>
<param name="port" value="22"/>
<param name="user" value="test"/>
<param name="auth_method" value="password"/>
<param name="command" value="uname -an"/>
<param name="password" value="super_secret"/>
</params>
<script language="java" java_class="sos.scheduler.job.SOSSSHJob2JSAdapter"/>
<run_time />
</job>
|
Related Downloads
- JobScheduler job_chain and jobs: download example
See also
This job