Page History

...

• Use Case:
  • Consider the situation where:
    
    • A job chain is to start at a certain point in time and after an arbitrary number of other job chains have been completed successfully.
    • The predecessor job chains run in an arbitrary sequence at arbitrary points in time.
  Example:
  • • parallel job chains
    • Subsequent jobs in started job chains check if jobs from parallel chains completed successfully.
  • Usually a split & sync pattern is used for similar use cases, however, synchronizing includes that both jobs have to be completed.
  • Instead the current use case is about check and control if a job of a parallel chain has completed without forcing both jobs to run synchronously
  • Job chain Z runs after 2 other job chains A and B have completed.

Solution

• Download: final_chain_job_chain_historycontrolParallelExecutionOfJobChainsExample.zip
• Extract the archive to a folder in your JobScheduler installation named ./config/live.
• The archive will extract the files to a sub-folder final_chain_job_chain_historycontrolParallelExecutionOfJobChainsExample. 
• Note that you can store the sample files in any folder you like.

Pattern

order job_chain_A1 [shape="ellipse",label="OrderJob Chain A\n start time: Friday 08:00 1\nhas no relevant dependencies",fillcolor="violetorange"]
orderjob_chain_B2 [shape="ellipse",label="OrderJob Chain B\n start time: Friday 07:00 2\nis started by Job Chain 1",fillcolor="violet"]
orderjob_chain_Z3 [shape="ellipse",label="OrderJob Chain Z\n start time: Friday 15:00 3\nis started by Job Chain 1",fillcolor="violetlightskyblue"]
 
job_chain_A4 [label="Job Chain A\nhas no relevant dependencies4\nis started by Job Chain 1",fillcolor="orangeaquamarine"]
job_chain_B5 [label="Job Chain B\nhas no relevant dependencies5\nis started by Job Chain 1",fillcolor="orangeorangered"]
job_chain_Z6 [label="Job Chain Z6\ndependsnis started onby theJob execution of\nJob Chain A and Job Chain B",Chain 4",fillcolor="yellow"]

split1 [label="Split 1", fillcolor="orange"]
job1 [label="Job 1", fillcolor="orange"]
job2 [label="Job 2", fillcolor="orange"]

job_check_predecessorsync1 [label="Sync 1", fillcolor="orange"]
job1_2 [label="Job 1.2", fillcolor="orange"]

job3 [label="Job check_predecessor3", fillcolor="lightskyblueviolet"]
job_next_jobjob4 [label="Job 4", fillcolor="Next Job"violet"]

job5 [label="Job 5", fillcolor="lightskyblue"]
check_predecessor_successfuljob6 [shape=diamond,label="execution successful\nfor Job Chain A and Job Chain B?",fillcolor="white"]
order_setback [label="Set back Order",fillcolor="white"]

order_A -> job_chain_A
order_B -> job_chain_B
order_Zlabel="Job 6", fillcolor="lightskyblue"]

job7 [label="Job 7", fillcolor="aquamarine"]
job8 [label="Job 8", fillcolor="aquamarine"]
job9 [label="Job 9", fillcolor="aquamarine"]

job10 [label="Job 10", fillcolor="orangered"]
job11 [label="Job 11", fillcolor="orangered"]
job12 [label="Job 12", fillcolor="orangered"]

job13 [label="Job 13", fillcolor="yellow"]

job_chain_1 -> split1
split1 -> job1 -> sync1
split1 -> job2 -> sync1
sync1 -> job1_2
 
job1 -> job_chain_Z
2 -> job3 -> job4
job2 -> job_chain_Z3 -> job5 -> job_check_predecessor
job_check_predecessorjob6

job1_2 -> job_chain_A
job_check_predecessor4 -> job7 -> job8 -> job9
job1_2 -> job_chain_B
job_check_predecessor5 -> job10 -> job11 -> job12

job9 -> job_chain_6 -> check_predecessor_successful
check_predecessor_successfuljob13
job12 -> job_next_job [label=" yes "]
check_predecessor_successful -> order_setback [label=" no "]
order_setback -> job_check_predecessorchain_6 [arrowhead="normal", color="lightslategrey"]

job4 -> job8 [arrowhead="normal", color="lightslategrey"]
job4 -> job11 [arrowhead="normal", color="lightslategrey"]
job6 -> job11 [arrowhead="normal", color="lightslategrey"]

Implementation

Components

• The solution contains three six job chains:
  • job_chain_A and job_chain_B represent the predecessor job chains that have no relevant dependencies with job_chain_Z.
  • job_chain_Z is the final job chain that checks if job_chain_A and job_chain_B have already been executed successfully.
• The solution implements a job check_predecessor_job_chain_Z that has been added at the start of job_chain_Z.
  • This job makes use of a parameter check_job_chains that is assigned a semicolon separated list of job chain names.
    • Job chain names can be specified with an absolute path (starting from the live folder) or with a path relative to the directory of this job.
    • Example:
      • check_job_chains = job_chain_A;job_chain_B
  • This job implements a spooler_process() function that reads the parameters and checks the job cahin history for successful execution of the job chains specified by the parameter.
  • Should all checks for previous successful execution provide a positive result then the current order will be moved to the next job chain node. Otherwise the current order is set back and will be repeated regularly according to the setback configuration for this job.

Usage

• • JobChain1 runs Job1 and Job2 in parallel.
    • On successful completion of Job1 then JobChain2 is run
    • On successful completion of Job2 then JobChain3 is run
  • JobChain2 runs Job3 and Job4
  • JobChain3 runs Job5 and Job6
  • JobChain4 runs after successful completion of JobChain1
    • JobChain4 runs Job7, Job8, Job9
    • Job7 runs 1st
    • Job8 runs once Job7 and Job4 have run successfully
  • JobChain5 runs after successful completion of JobChain1
    • JobChain5 runs Job10, Job11 and Job12
    • Job10 runs 1st
    • Job11 runs once Job10, Job4 and Job6 have run successfully
    • Job12 runs once Job11 has completed successfully
  • JobChain6 runs once Job9 and Job12 have completed successfully

Usage

• The first job split1 sets a number of parameters by use of a pre-processing Monitor for later control of successful completion of jobs. For each job to be controlled a pair of parameters is used:
  
  • The first parameter of the pair is a flag which is used for later checking if the dependent job is successfully completed.
    • The value of this parameter is initialised with "0" and is modified by the dependent jobs to "1".

    • In the above example the following parameters are used:

      • job4_successfully_completed=0 which will be checked by Job8 and Job11
      • job6_successfully_completed=0 which will be checked by Job11
      • job12_successfully_completed=0 which will be checked by Job13
  • The second parameter of the pair describes to which job chains the information about successful completion is broadcasted.
    • The example specifies the job chains for which a suspended order is continued if the current job completes successfully.
    • The current job chain  (here JobChain1) always is the first in the list.
      
    • The job chains can be specified by an absolute path or by a relative path to the current job chain.
      

    • In the above example the following values are used:

      • job4_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain5"

      • job6_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain5"

      • job12_commits_completion_to="/controlParallelExecutionOfJobChainsExample/JobChain1;JobChain4;JobChain6"

• The dependent jobs (here Job4, Job6 and Job12) use a post-processing monitor that implements commands to resume the orders
  
  • in all job chains which are specified by the commits_completion_to parameter
  • where the value of the successfully_completed parameter is set to "1".
• The jobs that wait for dependent jobs make use of a pre-processing monitor that checks the corresponding successfully_completed parameters.
  • These jobs use a task parameter dependent_jobs that specifies a list of dependent jobs.
  • In above example these are
    • Job8 : dependent_jobs="Job4"
    • Job11: dependent_jobs="Job4;Job6"
    • Job13: dependent_jobs="Job12"

• The above described pre- and post-processing monitors include the attached control_parallel_execution_of_job_chains.js JavaScript like this:

  Code Block
  title pre-processing monitor for split1 job collapse true
  <monitor name="set_params_for_check_jobs" ordering="1"> <script language="java:javascript"> <include live_file="control_parallel_execution_of_job_chains.js"/> <![CDATA[ function spooler_process_before() { //Specify the list of job chains for which Job4 commits that it has run successfully //These job chains can specify relative to the current job chain setParams4CheckingJobs( "Job4", ["Chain4", "Chain5"] ); //Specify the list of job chains for which Job6 commits that it has run successfully //These job chains can specify relative to the current job chain setParams4CheckingJobs( "Job6", ["Chain5"] ); //Specify the list of job chains for which Job12 commits that it has run successfully //These job chains can specify relative to the current job chain setParams4CheckingJobs( "Job12", ["Chain4", "Chain6"] ); return true; } ]]> </script> </monitor>

  Code Block
  title post-processing monitor for jobs Job4, Job6 and Job12 collapse true
  <monitor name="commit_successfully_run" ordering="0"> <script language="java:javascript"> <include live_file="control_parallel_execution_of_job_chains.js"/> <![CDATA[ function spooler_process_after(spooler_process_result) { return commit_successful_run(spooler_process_result); } ]]> </script> </monitor>

  Code Block
  title pre-processing monitor for job Job8, Job11 and Job13 collapse true
  <monitor name="check_dependent_jobs" ordering="0"> <script language="java:javascript"> <include live_file="control_parallel_execution_of_job_chains.js"/> <![CDATA[ function spooler_process_before() { return check_dependent_jobs(); } ]]> </script> </monitor>

• Positive Check
  • Add an order to job_chain_A using JOC's Add Order context menu.
    • The order should pass through the job chain without errors.
  • Add an order to job_chain_B using JOC's Add Order context menu.
    • The order should also pass through the job chain without errors.
  • Add an order to job_chain_Z using JOC's Add Order context menu.
    • The order should pass through the job chain without errors as all pre-conditions are met.
• Negative Check
• Modify the job do_something_job_chain_A to include an error, e.g. by adding the command exit 1 as the final line in the job script.
• Add an order to job_chain_A:
  
  • The order should proceed with an error.
• Add an order to job_chain_Z
  • The order should proceed with an error that is visible in the order log. This error will state that the predecessor job did not complete successfully.
  • The order will be set back and will be repeated according to the setback intervals that have been specified for the job check_predecessor_job_chain_Z.
Removing the modification to job do_something_job_chain_A and adding an order to job_chain_A that then runs successfully should result in a successful run of job_chain_Z - either by a new order added to job_chain_Z or by the next execution of the previous order that has been set back.

See also

• How to check the job chain history for previous job chain and order runs
• How to add a condition to check previous jobs before the execution of the current job chain

...