Page History

...

• Use Case:
  • Consider the situation where:
    
    • A job chain is to start at a number of parallel job chains
    • Subsequent jobs in started job chains check if jobs from parallel chains completed successfully.
  • Usually a split & sync pattern is used for similar use cases, however, synchronizing includes that both jobs have to be completed.
  • Instead the current use case is about check and control if a job of a parallel chain has completed without forcing both jobs to run synchronously.

Solution

• Download: final_chain_job_chain_historycontrolParallelExecutionOfJobChainsExample.zip
• Extract the archive to a folder in your JobScheduler installation named ./config/live.
• The archive will extract the files to a sub-folder parallel_job_chain_controlcontrolParallelExecutionOfJobChainsExample. 
• Note that you can store the sample files in any folder you like.

...

 job_chain_1 [label="Job Chain 1\nhas no relevant dependencies",fillcolor="orange"]
job_chain_2 [label="Job Chain 2\nis started by Job Chain 1",fillcolor="violet"]
job_chain_3 [label="Job Chain 3\nis started by Job Chain 1",fillcolor="lightskyblue"]
job_chain_4 [label="Job Chain 4\nis started by Job Chain 1",fillcolor="aquamarine"]
job_chain_5 [label="Job Chain 5\nis started by Job Chain 1",fillcolor="orangered"]
job_chain_6 [label="Job Chain 6\nis started by Job Chain 4",fillcolor="yellow"]

split1 [label="Split 1", fillcolor="orange"]
job1 [label="Job 1", fillcolor="orange"]
job2 [label="Job 2", fillcolor="orange"]
sync1 [label="Sync 1", fillcolor="orange"]
job1_2 [label="Job 1.2", fillcolor="orange"]

job3 [label="Job 3", fillcolor="violet"]
job4 [label="Job 4", fillcolor="violet"]

job5 [label="Job 5", fillcolor="lightskyblue"]
job6 [label="Job 6", fillcolor="lightskyblue"]

job7 [label="Job 7", fillcolor="aquamarine"]
job8 [label="Job 8", fillcolor="aquamarine"]
job9 [label="Job 9", fillcolor="aquamarine"]

job10 [label="Job 10", fillcolor="orangered"]
job11 [label="Job 11", fillcolor="orangered"]
job12 [label="Job 12", fillcolor="orangered"]

job13 [label="Job 13", fillcolor="yellow"]

subgraph chain1 {
  color=orange;  
  style=filled;
  fontsize = 40;
  
  label="Job Chain 1\nhas no relevant dependencies";
  split1->job1->sync1;
  split1->job2->sync1;
  sync1->job1_2;
}

subgraph chain2 {
  color=violet;  
  style=filled;
  
  label="Job Chain 2\nis started by Job Chain 1";
  job3->job4;
}

chain1->chain2;


job_chain_1 -> split1
split1 -> job1 -> sync1
split1 -> job2 -> sync1
sync1 -> job1_2
 
job1 -> job_chain_2 -> job3 -> job4
job2 -> job_chain_3 -> job5 -> job6

job1_2 -> job_chain_4 -> job7 -> job8 -> job9
job1_2 -> job_chain_5 -> job10 -> job11 -> job12

job9 -> job_chain_6 -> job13
job12 -> job_chain_6 [arrowhead="normal", color="lightslategrey"]

job4 -> job8 [arrowhead="normal", color="lightslategrey"]
job4 -> job11 [arrowhead="normal", color="lightslategrey"]
job6 -> job11 [arrowhead="normal", color="lightslategrey"]

...

• The solution contains six job chains:
  • JobChain1 runs Job1 and Job2 in parallel.
    • On successful completion of Job1 then JobChain2 is run
    • On successful completion of Job2 then JobChain3 is run
  • JobChain2 runs Job3 and Job4
  • JobChain3 runs Job5 and Job6
  • JobChain4 runs after successful completion of JobChain1
    • JobChain4 runs Job7, Job8, Job9
    • Job7 runs 1st
    • Job8 runs once Job7 and Job4 have run successfully
  • JobChain5 runs after successful completion of JobChain1
    • JobChain5 runs Job10, Job11 and Job12
    • Job10 runs 1st
    • Job11 runs once Job10, Job4 and Job6 have run successfully
    • Job12 runs once Job11 has completed successfully
  • JobChain6 runs once Job9 and Job12 have completed successfully

Usage

• The first job split1 set some pairs sets a number of parameters in by use of a pre-processing to control which job tells that it is successfully completed. Monitor for later control of successful completion of jobs. For each job to be controlled a pair of parameters is used:
  
  • The first parameter of the pair is a flag which is used for a later check checking if the dependent jobs are job is successfully completed.
    •  The value of this parameter is initialised with "0" and will be is modified by the dependent jobs with to "1".

    • In the above example these the following parameters are used:

      • job4_successfully_completed=0 which will be checked by Job8 and Job11
      • job6_successfully_completed=0 which will be checked by Job11
      • job12_successfully_completed=0 which will be checked by Job13
  • The second parameter of the pair describes to which job commits which job chains that it has successfully completed. job chains the information about successful completion is broadcasted.
    • The example specifies the job chains for which a suspended order is continued if the current job completes successfully.
    • You need to specify all job chains, in which the order can be located with respect to the checking jobs
    • The current job chain  (here JobChain1) always is always the first in the list.
      
    • The job chains can specify with its be specified by an absolute path or by a relative path to the current job chain.
      

    • In the above example these the following values are used:

      • job4_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain5"

      • job6_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain5"

      • job12_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain6"

• The dependent jobs (here Job4, Job6 and Job12) have use a post-processing where they monitor that implements commands to resume the orders
  
  • in all job chains which are specified in by the "commits_completion_to" parameter parameter
  • where the value of the "successfully_completed" parameter  parameter is set to "1".
• The jobs which that wait of its for dependent jobs have make use of a pre-processing where they check monitor that checks the corresponding "successfully_completed" parameters parameters.
  • These jobs have use a task parameter "dependent_jobs" where  that specifies a list of its dependent jobs are specified.
  • In above example these are
    • Job8 : dependent_jobs="Job4"
    • Job11: dependent_jobs="Job4;Job6"
    • Job13: dependent_jobs="Job12"

• The above described pre- and post-processings processing monitors include the attachedattached control_parallel_execution_of_job_chains.js file so that you have following small configuration JavaScript like this:

  Code Block
  title pre-processing of monitor for split1 job collapse true
  <monitor name="set_params_for_check_jobs" ordering="1"> <script language="javax.scriptjava:ecmascriptjavascript"> <include live_file="control_parallel_execution_of_job_chains.js"/> <![CDATA[ function spooler_process_before() { //Specify the list of job chains for which Job4 commits that it has run successfully //These job chains can specify relative to the current job chain setParams4CheckingJobs( "Job4", ["Chain4", "Chain5"] ); //Specify the list of job chains for which Job6 commits that it has run successfully //These job chains can specify relative to the current job chain setParams4CheckingJobs( "Job6", ["Chain5"] ); //Specify the list of job chains for which Job12 commits that it has run successfully //These job chains can specify relative to the current job chain setParams4CheckingJobs( "Job12", ["Chain4", "Chain6"] ); spooler_log.info(spooler_task.order().params().xml()); return true; } ]]> </script> </monitor>

  Code Block
  title post-processing of monitor for jobs Job4, Job6 and Job12 collapse true
  <monitor name="commit_successfully_run" ordering="0"> <script language="javax.scriptjava:ecmascriptjavascript"> <include live_file="control_parallel_execution_of_job_chains.js"/> <![CDATA[ function spooler_process_after(spooler_process_result) { return commit_successful_run(spooler_process_result); } ]]> </script> </monitor>

  Code Block
  title pre-processing of monitor for job Job8, Job11 and Job13 collapse true
  <monitor name="check_dependent_jobs" ordering="0"> <script language="javax.scriptjava:ecmascriptjavascript"> <include live_file="control_parallel_execution_of_job_chains.js"/> <![CDATA[ function spooler_process_before() { return check_dependent_jobs(); } ]]> </script> </monitor>

...