Page History

Scenario

In this example user we want to poll for an incoming file on the ServerAServer A, once file(s) matching with the a reguler expression is are found on ServerA, Server A then JobScheduler should transfer the file(s) from ServerA Server A to ServerBServer B.
After successfully successful transfer of file on ServerB, JobScheduler should start an processing script on ServerB files to Server B then JobScheduler should execute a script for post-processing on Server B via SSH.

digraph "Example: Server to Server Transfer with polling" \{

rankdir=LR;

edge [
color="#31CEF0"
];

source_server [label="DataServerA\nTransfer Source (FTP)"];
target_server  [label="DataServerB\nTransfer Target (sFTPSFTP)"];
jobscheduler  [label="JobScheduler \n JADEYADE JITL job \n SSH JITL Job"];


source_server->target_server [label="2. datafile transfer"];
source_server->jobscheduler  [dir=both label="commands" color="grey"];
jobscheduler->source_server  [dir=right label="1. polling" color="blue"];
jobscheduler->target_server  [dir=both label="commands"  color="grey"];
jobscheduler->target_server  [dir=righ label="3. ssh command"  color="red"];
}

Solution

• For such a file transfer and processing workflow

...

• JobScheduler has built-in JITL jobs, namely

...

• the YADE job and SSH job.
• The

...

• YADE JITL Jobs be configured to transfer data from Server-to-Server without

...

• touchdown, more information about Server-to-Server transfer can be found here: Example for YADE Server-to-Server File Transfer.

...

• YADE has a built-in polling feature

...

• that can be configured by using polling

...

• parameters, i.e.
  • poll_interval: frequency of polling in seconds and
  • poll_timeout: maximum

...

• • duration in seconds for polling.
• The job

...

• chain can be configured by an order to run at a predefined time, to repeat execution or

...

• to be started manually.

Job Chain: FileTransferAndProcessing.job_chain.xml

The job chain FileTransferAndProcessing has two nodes:

1. jade_s2s_file_transfer: for server to server data transfer and
2. ssh_start_proccessing: starting the processing script using ssh
    

<?xml version="1.0" encoding="ISO-8859-1"?>


<job_chain  orders_recoverable="yes" visible="yes" title="Tranasfer Data from ServerA to ServerB and start Processing on ServerB vis SSH" max_orders="1" name="DataTransferProcessing">

    <job_chain_node  state="jade_s2s_file_transfer" job="jade_s2s_file_transfer" error_state="error" next_state="ssh_start_proccessing" on_error="suspend"/>

    <job_chain_node  state="ssh_start_proccessing" job="ssh_start_proccessing" next_state="success" error_state="error" on_error="suspend"/>

    <job_chain_node  state="success"/>

    <job_chain_node  state="error"/>

</job_chain>

Step 1: Job jade_s2s_file_transfer

...

• This job is

...

• a YADE JITL job

...

• that is configured to poll a source

...

• host (

...

• via FTP) for

...

• files matching file_

...

• spec=^TEST*\.log$ for every 30 seconds

...

• during a maximum of 600 seconds.

...

• The matching file(s)

...

• will be

...

• copied to the target

...

• host (

...

• SFTP).

...

• More information about

...

• YADE JITL job configuration can be found here

...

• : JITL Jobs by topic.

Code Block
language html/xml xml title Job jade_s2s_file_transfer collapse true
<?xml version="1.0" encoding="ISO-8859-1"?> <job title="S2S transfer with polling (API Job for JobScheduler Advanced Data Exchange)" order="yes" stop_on_error="no" name="jade_s2s_file_transfer"> <description > <include file="jobs/jadeJob.xml"/> </description> <params > <param name="file_spec" value="\.log$"/> <param name="operation" value="copy"/> <param name="verbose" value="9"/> <param name="poll_interval" value="30"/> <param name="poll_minfiles" value="1"/> <param name="poll_timeout" value="600"/> <param name="source_protocol" value="ftp"/> <param name="source_host" value="mp.sos"/> <param name="source_user" value="test"/> <param name="source_password" value="super_secret"/> <param name="source_dir" value="/outbound"/> <param name="target_protocol" value="sftp"/> <param name="target_host" value="homer.sos"/> <param name="target_user" value="test"/> <param name="target_password" value="super_secret"/> <param name="target_ssh_auth_method" value="password"/> <param name="target_dir" value="/home/test/inbound"/> </params> <script language="java" java_class="sos.scheduler.jade.JadeJob"/> <run_time /> </job>

Step 2: Job ssh_start_proccessing

...

• This job is

...

• a JITL job

...

• that is configured to execute

...

• a command on a remote

...

• host using

...

• SSH. The SSH JITL job can be configured to connect using

...

• a public/

...

• private key pair

...

• or with

...

• user/password authentication.
• More information about SSH JITL job configuration can be found here: SSH_JITL_JOB_PARAMETERS. 

Code Block
language html/xml xml title Job ssh_start_proccessing collapse true
<?xml version="1.0" encoding="ISO-8859-1"?> <job title="Launch commands or executable files by SSH" order="yes" name="ssh_start_proccessing"> <description > <include file="jobs/SOSSSHJob2JSAdapter.xml"/> </description> <params > <param name="host" value="homer.sos"/> <param name="port" value="22"/> <param name="user" value="test"/> <param name="auth_method" value="password"/> <param name="command" value="uname -an"/> <param name="password" value="super_secret"/> </params> <script language="java" java_class="sos.scheduler.job.SOSSSHJob2JSAdapter"/> <run_time /> </job>

Also see

1. JADE S2S implementation details
2. Server to Server Transfer
3. Server to Server Transfer with Polling

Download example

Related Downloads

• JobScheduler job_chain /and jobs: download example

<mscgen>
// Remote configuration
msc {

 width"800";

config_server labeh1. "Configuration Server JobScheduler",workload label"Workload JobScheduler";

...

See also

• YADE Implementation Architecture
• How to transfer files with YADE from Server A to Server B using polling and post-processing
• Example for YADE Server-to-Server File Transfer
• Example for YADE Server-to-Server File Transfer with polling

config_server h1. workload label "First Request HTTP using JETTY Port)";

  config_server box config_server[labeh1. "Register Client", textbgcolour"#7f7fff",  textcolouh1. "#FFFFFF"];
  config_server < workload  [labelh1. "Request every 15 minutes (HTTP using JETTY Port)"];
  config_server> workload  [labelh1. "Sending configuration changes (HTTP using JETTY Port)"];
    ---  [label "loop every 15 minutes", ID="*"];

}
</mscgen>