Page History

Scope

• Use Case
  • Run parts of a job chain in parallel, i.e. split jobs for parallel processing.
  • join up Synchronize jobs after parallel processing.
• Solution Outline
  • Two standard JITL Jobs are provided for this:
    • split an incoming order into multiple orders that are running in parallel through a job chain within a range of predefined job chain nodes.
    • joinsync the orders that have previously been split in a job chain node, i.e. this job forces any split orders to complete to this job chain node before processing continues.
• References
  • Example showing how to set up a splitter job
  • Example for parallel execution

Solution

• Download Download split_joinsync.zip
• Extract the archive to a folder ./config/live/issues of your JobScheduler installation.
• The archive will extract the files to a folder split_joinsync. 
• You can store the sample files to a any folder as you like, the solution does not make use of specific folder names or job names.

Pattern

job_chain [label="Job Chain",fillcolor="orange"]
job_1 [label="Job 1",fillcolor="lightskyblue"]
job_2 [label="Job 2",fillcolor="lightskyblue"]
job_split [label="Job Split",fillcolor="lightskyblue"]
job_3 [label="Job 3",fillcolor="lightskyblue"]
job_3a [label="Job 3a",fillcolor="lightskyblue"]
job_4 [label="Job 4",fillcolor="lightskyblue"]
job_4a [label="Job 4a",fillcolor="lightskyblue"]
job_5 [label="Job 5",fillcolor="lightskyblue"]
job_joinsync [label="Job JoinSynchronize",fillcolor="lightskyblue"]

job_chain -> job_1
job_1 -> job_2
job_2 -> job_split
job_split -> job_3
job_split -> job_4
job_3 -> job_3a -> job_joinsync
job_4 -> job_4a -> job_joinsync
job_joinsync -> job_5

Implementation

Components

• The job chain and jobs job1 to job5 provided by the sample are not specific specifiic for this solution, they represent most simple shell scripts.
• The job split_partitions is the Splitter JITL Job provided with the Java class com.sos.jitl.splitter.JobChainSplitterJSAdapterClass.
  • The job is used with the following parameters:
    • state_names: a list of semicolon separated job node states to which split splitted orders are added. For each entry in this list an individual order is created.
      • The state names correspond to the states that the respective job nodes are associated with in the job chain definition.
      • In order to support better graphical output for the JOE graphical diagram the state names are prefixed with the state of the split_partitions job (corresponding to its name), a colon and the name of the job associated with the state. Example for state of job3: split_partitions:job3
      • Example for state_names parameter value: split_partitions:job3;split_partitions:job4
    • joinsync_state_name: this parameter is required exclusively for improved graphical output from JOE. It accepts the value of the state that is associated with the joinsync_partitions job. node 
  • Any number of jobs can follow the jobs that are referenced by the state_names parameter.
• The job job_chain1.joinsync_partitions is used to join up split synchronize splitted orders and is provided by the Join Orders Sync JITL Job with the Java class com.sos.jitl.joinsync.JobSchedulerJoinOrdersJSAdapterClassJobSchedulerSynchronizeJobChainsJSAdapterClass.
  • The recommended naming convention for this job includes to use the job chain name as a prefix: due to the nature of this job to join up synchronize across all job chains that are running in the system and that make use of the same join sync job name you should prefix the job name with the job chain name if you want this job to be limited to work for the current job chain.
  • This job is used without parameters.

Usage

• Add an order to job chain job_chain1 by use of JOC.
• Consider the processing that would 
  • split the execution into two subsequent orders that run for job3->job3a and job4->job4a. 
  • move the current order to the joinsync_partitions job node.
• The processing for job3->job3a and job4->job4a will require different execution time. All orders will wait in the joinsync_partitions job node for any split splitted order to arrive. With all split splitted orders being completed the processing will continue with job5.

...