How to control parallel execution of job chains

Scope

Use Case:
- Consider the situation where:
  - A job chain is to start at a number of parallel job chains
  - Subsequent jobs in started job chains check if jobs from parallel chains completed successfully.
- Usually a split & sync pattern is used for similar use cases, however, synchronizing includes that both jobs have to be completed.
- Instead the current use case is about check and control if a job of a parallel chain has completed without forcing both jobs to run synchronously.

Solution

Download: controlParallelExecutionOfJobChainsExample.zip
Extract the archive to a folder in your JobScheduler installation named ./config/live.
The archive will extract the files to a sub-folder parallel_job_chain_control.
Note that you can store the sample files in any folder you like.

Pattern

Implementation

Components

The solution contains six job chains:
- JobChain1 runs Job1 and Job2 in parallel.
  - On successful completion of Job1 then JobChain2 is run
  - On successful completion of Job2 then JobChain3 is run
- JobChain2 runs Job3 and Job4
- JobChain3 runs Job5 and Job6
- JobChain4 runs after successful completion of JobChain1
  - JobChain4 runs Job7, Job8, Job9
  - Job7 runs 1st
  - Job8 runs once Job7 and Job4 have run successfully
- JobChain5 runs after successful completion of JobChain1
  - JobChain5 runs Job10, Job11 and Job12
  - Job10 runs 1st
  - Job11 runs once Job10, Job4 and Job6 have run successfully
  - Job12 runs once Job11 has completed successfully
- JobChain6 runs once Job9 and Job12 have completed successfully

Usage

The first job split1 sets a number of parameters by use of a pre-processing Monitor for later control of successful completion of jobs. For each job to be controlled a pair of parameters is used.
- The first parameter of the pair is a flag which is used for later checking if the dependent job is successfully completed.
  - The value of this parameter is initialised with "0" and is modified by the dependent jobs to "1".
  - In the above example the following parameters:
    - job4_successfully_completed=0 which will be checked by Job8 and Job11
    - job6_successfully_completed=0 which will be checked by Job11
    - job12_successfully_completed=0 which will be checked by Job13
- The second parameter of the pair describes to which job chains the information about successful completion is broadcasted.
  - The example specifies the job chains for which a suspended order is continued if the current job completes successfully.
  - The current job chain (here JobChain1) is always the first in the list.
  - The job chains can be specified by an absolute path or a relative path to the current job chain.
  - In the above example these are
    - ```
    job4_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain5"
```
- ```
job6_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain5"
```
    - ```
    job12_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain6"
```
The dependent jobs (here Job4, Job6 and Job12) use a post-processing monitor that implements commands to resume the orders
- in all job chains which are specified by the commits_completion_to parameter
- where the value of the successfully_completed parameter is set to "1"
The jobs that wait for dependent jobs make use of a pre-processing monitor that checks the corresponding successfully_completed parameters.
- These jobs use a task parameter dependent_jobs that specifies a list of dependent jobs.
- In above example these are
  - Job8 : dependent_jobs="Job4"
  - Job11: dependent_jobs="Job4;Job6"
  - Job13: dependent_jobs="Job12"

The above described pre- and post-processing monitors include the attached control_parallel_execution_of_job_chains.js JavaScript like this:

pre-processing of split1

<monitor  name="set_params_for_check_jobs" ordering="1">
        <script  language="java:javascript">
            <include  live_file="control_parallel_execution_of_job_chains.js"/>
            <![CDATA[
function spooler_process_before() {
    
    //Specify the list of job chains for which Job4 commits that it has run successfully
    //These job chains can specify relative to the current job chain
    
    setParams4CheckingJobs( "Job4", ["Chain4", "Chain5"] );

    //Specify the list of job chains for which Job6 commits that it has run successfully
    //These job chains can specify relative to the current job chain
    
    setParams4CheckingJobs( "Job6", ["Chain5"] );    
    
    //Specify the list of job chains for which Job12 commits that it has run successfully
    //These job chains can specify relative to the current job chain
    
    setParams4CheckingJobs( "Job12", ["Chain4", "Chain6"] );
    
    return true;
}
            ]]>
        </script>
    </monitor>

post-processing of Job4, Job6 and Job12

<monitor  name="commit_successfully_run" ordering="0">
        <script  language="java:javascript">
            <include  live_file="control_parallel_execution_of_job_chains.js"/>
            <![CDATA[
function spooler_process_after(spooler_process_result) {
    return commit_successful_run(spooler_process_result);
}
            ]]>
        </script>
    </monitor>

pre-processing of Job8, Job11 and Job13

<monitor  name="check_dependent_jobs" ordering="0">
        <script  language="java:javascript">
            <include  live_file="control_parallel_execution_of_job_chains.js"/>
            <![CDATA[
function spooler_process_before() {
    return check_dependent_jobs();
}
            ]]>
        </script>
    </monitor>

Space shortcuts

Page tree