...
- The JobScheduler Universal Agent can be used to watch incoming files and to trigger a job chain start for each file.
- The mechanisms for starting job chains apply as stated in the section about File Watching
- Subsequent jobs can be executed on the JobScheduler Master or on any Agent involved.
- Any number of jobs can be executed in sequence or in parallel for incoming files.
- Incoming files can be removed or moved to a different location
- by any of the jobs involved or
- at the end of processing by a file order sink that is specified with the job chain.
- Should an incoming file be treated by jobs, e.g. parsed or otherwise accessed, then the file has to be transferred to the host on which the respective job is executed.
- Use Cases
Configuration
- File watching is configured by the JobScheduler Master. No configuration is required on the JobScheduler Agent.
- The JobScheduler Master holds the configuration of a job chain with a file order source that is assigned to the Agent.
...
Code Block |
---|
language | xml |
---|
title | Process Class agent_in_dmz.process_class.xml |
---|
collapse | true |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<process_class max_processes="10" remote_scheduler="http://dmzhost:4445">
</process_class> |
Code Block |
---|
language | xml |
---|
title | Job Chain remote_files_local_processing |
---|
collapse | true |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job_chain file_watching_process_class="agent_in_dmz">
<file_order_source directory="/srv/files/1/in"/>
<job_chain_node state="start" job="job1" next_state="continue" error_state="error"/>
<job_chain_node state="continue" job="job2" next_state="success" error_state="error"/>
<file_order_sink state="success" move_to="/tmp/jobscheduler/file/success" remove="no"/>
<file_order_sink state="error" move_to="/tmp/jobscheduler/file/error" remove="no"/>
</job_chain> |
...
The following configuration applies to a job chain that specifies file watching and job execution on the Agent:
Code Block |
---|
language | xml |
---|
title | Job Chain remote_files_remote_processing |
---|
collapse | true |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job_chain file_watching_process_class="agent_in_dmz" process_class="agent_in_dmz">
<file_order_source directory="/srv/files/2/in"/>
<job_chain_node state="start" job="job1" next_state="continue" error_state="error"/>
<job_chain_node state="continue" job="job2" next_state="success" error_state="error"/>
<file_order_sink state="success" move_to="/tmp/jobscheduler/file/success" remove="no"/>
<file_order_sink state="error" move_to="/tmp/jobscheduler/file/error" remove="no"/>
</job_chain> |
...
- A process class is configured in a separate file, e.g.
agent_in_dmz.process_class.xml,
as stated with the above sample. The process class specifies the protocol, host and port that the Agent is operated for. - The job chain references the above process class that causes the subsequent configuration for file order sources, file order sinks and jobs to be applied to the respective JobScheduler Agent.
- The
<file_order_source>
is configured as explained for a JobScheduler Master and applies to the JobScheduler Agent. - Subsequent jobs are executed on the JobScheduler Agent.
- The
<file_order_sink>
specifies incoming files to be moved to different directories on the JobScheduler Agent host depending on the execution result of the job chain. - For more detailed explanations see see:
Jira |
---|
server | SOS JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 6dc67751-9d67-34cd-985b-194a8cdc9602 |
---|
key | JS-1301 |
---|
|
Jira |
---|
server | SOS JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 6dc67751-9d67-34cd-985b-194a8cdc9602 |
---|
key | JS-1300 |
---|
|
Jira |
---|
server | SOS JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 6dc67751-9d67-34cd-985b-194a8cdc9602 |
---|
key | JS-1422 |
---|
|
File watching with file transfer for job execution on the JobScheduler Master
...
Code Block |
---|
language | xml |
---|
title | Job Chain remote_files_local_transfer |
---|
collapse | true |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job_chain file_watching_process_class="agent_in_dmz">
<file_order_source directory="/srv/files/3/in"/>
<job_chain_node state="transfer" job="jade" next_state="success" error_state="error"/>
<job_chain_node state="success"/>
<file_order_sink state="error" move_to="/tmp/jobscheduler/file/error" remove="no"/>
</job_chain> |
...
Code Block |
---|
language | xml |
---|
title | File Transfer Job |
---|
collapse | trueJob jade |
---|
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<job title="API Job for JobScheduler Advanced Data Exchange" order="yes" stop_on_error="no" name="jade">
<description >
<include file="jobs/jadeJob.xml"/>
</description>
<params >
<param name="operation" value="move"/>
<param name="source_host" value="dmzhost"/>
<param name="source_protocol" value="sftp"/>
<param name="source_ssh_auth_method" value="password"/>
<param name="source_user" value="foo"/>
<param name="source_password" value="bar"/>
<param name="target_dir" value="c:\temp"/>
<param name="target_protocol" value="local"/>
<param name="file_path" value="%scheduler_file_path%"/>
</params>
<script language="java" java_class="sos.scheduler.jade.JadeJob"/>
<run_time />
</job> |
...
- A process class is configured in a separate file, e.g.
agent_in_dmz.process_class.xml,
as stated with the above sample. The process class specifies the protocol, host and port that the Agent is operated for. - The job chain references the above process class with the
file_watching_process_class
attribute that causes the subsequent configuration for file order sources to be applied to the respective JobScheduler Agent. - The
<file_order_source>
is configured as explained for a JobScheduler Master and applies to the JobScheduler Agent. - The JADE YADE file transfer Job is executed on the JobScheduler Master
- It receives the triggered file, specified in the
file_path
parameter with %scheduler_file_path%
, from the Agent host.- At runtime
%scheduler_file_path%
is substituted with the actual path of the triggered file.
- Alternatively, a JADE YADE job could be run on JobScheduler Universal Agent to send the file to the JobScheduler Master host.
- If the transfer is successfullsuccessful, a
file_order_sink
is not required, as the JADE YADE Job is configured with the move
command which removes the input file from the agent host after transfer.
References
Change Management References
Jira |
---|
server | SOS JIRA |
---|
columns | type,key,issuelinks,fixversions,status,priority,summary,updated |
---|
maximumIssues | 20 |
---|
jqlQuery | labels in (file-order-source,file-order-sink) and labels in (process-class) |
---|
serverId | 6dc67751-9d67-34cd-985b-194a8cdc9602 |
---|
|
See also