How to control parallel execution of job chains

Scope

Use Case:
- Consider the situation where:
  - A job chain is to start at a number of parallel job chains
  - Subsequent jobs in started job chains check if jobs from parallel chains completed successfully.
- Usually a split & sync pattern is used for similar use cases, however, synchronizing includes that both jobs have to be completed.
- Instead the current use case is about check and control if a job of a parallel chain has completed without forcing both jobs to run synchronously.

Solution

Download: final_chain_job_chain_history.zip
Extract the archive to a folder in your JobScheduler installation named ./config/live.
The archive will extract the files to a sub-folder parallel_job_chain_control.
Note that you can store the sample files in any folder you like.

Pattern

Implementation

Components

The solution contains six job chains:
- JobChain1 runs Job1 and Job2 in parallel.
  - On successful completion of Job1 then JobChain2 is run
  - On successful completion of Job2 then JobChain3 is run
- JobChain2 runs Job3 and Job4
- JobChain3 runs Job5 and Job6
- JobChain4 runs after successful completion of JobChain1
  - JobChain4 runs Job7, Job8, Job9
  - Job7 runs 1st
  - Job8 runs once Job7 and Job4 have run successfully
- JobChain5 runs after successful completion of JobChain1
  - JobChain5 runs Job10, Job11 and Job12
  - Job10 runs 1st
  - Job11 runs once Job10, Job4 and Job6 have run successfully
  - Job12 runs once Job11 has completed successfully
- JobChain6 runs once Job9 and Job12 have completed successfully

Usage

The first job split1 set some pairs of parameters in a pre-processing to control which job tells that it is successfully completed.
- The first parameter of the pair is a flag which is used for a later check if the dependent jobs are successfully completed.
  - The value of this parameter is initialised with "0" and will be modified by the dependent jobs with "1".
  - In above example these are
    - job4_successfully_completed=0 which will be checked by Job8 and Job11
    - job6_successfully_completed=0 which will be checked by Job11
    - job12_successfully_completed=0 which will be checked by Job13
- The second parameter of the pair describes which job commits which job chains that it has successfully completed.
  - You need to specify all job chains, in which the order can be located with respect to the checking jobs
  - The current job chain (here JobChain1) is always the first in the list.
  - The job chains can specify with its absolute path or relative to the current job chain.
  - In above example these are
    - ```
    job4_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain5"
```
- ```
job6_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain5"
```
    - ```
    job12_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain6"
```
The dependent jobs (here Job4, Job6 and Job12) have a post-processing where they resume the orders
- in all job chains which are specified in the "commits_completion_to" parameter
- where the value of the "successfully_completed" parameter is set to "1"
The jobs which wait of its dependent jobs have a pre-processing where they check the corresponding "successfully_completed" parameters
- These jobs have a task parameter "dependent_jobs" where a list of its dependent jobs are specified.
- In above example these are
  - Job8 : dependent_jobs="Job4"
  - Job11: dependent_jobs="Job4;Job6"
  - Job13: dependent_jobs="Job12"

The above described pre- and post-processings include the attached control_parallel_execution_of_job_chains.js file so that you have following small configuration

pre-processing of split1

<monitor  name="set_params_for_check_jobs" ordering="1">
        <script  language="javax.script:ecmascript">
            <include  live_file="control_parallel_execution_of_job_chains.js"/>
            <![CDATA[
function spooler_process_before(){
    
    //Specify the list of job chains for which Job4 commits that it has run successfully
    //These job chains can specify relative to the current job chain
    
    setParams4CheckingJobs( "Job4", ["Chain4", "Chain5"] );

    
    //Specify the list of job chains for which Job6 commits that it has run successfully
    //These job chains can specify relative to the current job chain
    
    setParams4CheckingJobs( "Job6", ["Chain5"] );
    
    
    //Specify the list of job chains for which Job12 commits that it has run successfully
    //These job chains can specify relative to the current job chain
    
    setParams4CheckingJobs( "Job12", ["Chain4", "Chain6"] );

    
    spooler_log.info(spooler_task.order().params().xml());    
    return true;
}
            ]]>
        </script>
    </monitor>

post-processing of Job4, Job6 and Job12

<monitor  name="commit_successfully_run" ordering="0">
        <script  language="javax.script:ecmascript">
            <include  live_file="control_parallel_execution_of_job_chains.js"/>
            <![CDATA[
function spooler_process_after(spooler_process_result){
    return commit_successful_run(spooler_process_result);
}
            ]]>
        </script>
    </monitor>

pre-processing of Job8, Job11 and Job13

<monitor  name="check_dependent_jobs" ordering="0">
        <script  language="javax.script:ecmascript">
            <include  live_file="control_parallel_execution_of_job_chains.js"/>
            <![CDATA[
function spooler_process_before(){
    return check_dependent_jobs();
}
            ]]>
        </script>
    </monitor>

Space shortcuts

Page tree