Product Knowledge Base

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Deutsche Version

Example: Multiple parallel processes in a job chain

Specification:

  • The job chain is to start with a job named "truncate_export_table".
  • After this job has been completed four jobs named "table partition" are to be run in parallel.
  • A single job that indexes the new partition tables is then to run.
  • Finally, a further four jobs that test the individual partition tables are to start in parallel.
    This job chain is shown schematically in the diagram in the "Diamond" section below.

Writing the job chain

The following steps have to be followed to realize a job chain that meets the requirements listed above:

  1. A "splitter" job has to be included for each "series" of job nodes that are to be processed in parallel. The splitter generates an order for each series of parallel job nodes starts these orders as soon as it itself has been started.
  2. In order to do this the splitter job has to "know" the names of the states corresponding to the first node of each job node series. These names are specified in the splitter job's state_names parameter (see Setting_parameters).
  3. The parallel processing normally ends at a specific node in the the chain with processing then continuing serially. This node is the synchronisation node and is implemented using the Sync-Job.

The "Diamond" job chain structure

The example job chain will look like this (the diagram has been generated with the Sandbox JOE Version):

We refer to the pattern that results with this type of job chain as a "diamond" pattern. Such patterns can occur more than once in a job chain: both sequentially, as shown in the diagram above, in parallel and nested. They can also be combined with other job chain patterns such as emerald or cross-over patterns (see Synchronisation_of_Job_Chains).

Job chain list view

The next illustration shows a list view of the job chain as produced by JOE:

The "Splitter" job

A generic splitter job is delivered with the JobScheduler JITL jobs. This job can be found in the "./live/sos/jitl" directory.

We recommend that the following syntax is used for the names of job nodes that are to be processed in parallel:

  • "splitter job node name" ":" "job name". In the example diagram above, one of the first nodes would then have the name split_partitions:partition_1.
    Although this syntax is not necessary for the correct functioning of the JobScheduler, it allows the diagram algorithm in JOE (that was used to draw the "diamond" diagram shown above) to know and correctly display the nodes that directly follow on from the splitter. It is necessary to use this type of job name syntax as the syntax used by JobScheduler does not recognize predecessor relationships (only successors).

The use of the above syntax is not necessary for the correct functioning of the JobScheduler.

Splitter job parameters

See documentation of the JobChainSplitter job.

The state_names parameter

  • The splitter job state_names parameter is used to specify the node names of the jobs that are to be first started in parallel (see Setting_parameters).
  • The node names are to be seperated by semi-colons.
  • In jobs chains with this diamond pattern structure, the parameters are specified for the job chain and referred to as node parameters. Node parameters can be used to specify parameters for more than one splitter in a job chain, independently of one another, as in our example, without creating conflicts.

The parameters for the split_partitions splitter job - as shown in JOE - are:

The "Sync" job

A unique sync job is required at the end of every group of processes running in parallel (see Setting_up_a_sync_job) when the nodes in the job chain after the sync node are only to be processed after all the jobs (tasks) that are to be carried out in parallel have been completed without errors.
Each sync job has to be unique within a JobScheduler instance - and within a job chain - as long as a cross-over pattern has not been implemented (see Synchronizaton_of_Job_Chains).

For more information see the documentation for the JobSchedulerSynchronizeJobChains job.

Best practice

Use "Start" and "End" nodes:

We recommend that you use our /sos/jitl/JobChainStart start job as the first node in every job chain and our /sos/jitl/JobChainEnd end job as the last full node.

Give each sync job a unique name:

Give each sync job a unique name by using the name of the job chain in which the sync job is included as a prefix in the name of the sync job.

  • For example: ideal_insert_to_export_table_parallel.sync_partitions

Follow our convention for node naming:

Splitter nodes

We recommend that you use the following syntax for the names of job nodes that are processed in parallel:

  • "splitter job node name" ":" "job name". In the example diagram above, one of the first nodes would then have the name split_partitions:partition_1.
    This syntax allows the diagram algorithm in JOE to draw job chain diagrams and correctly display the nodes that directly follow on from the splitter. It is necessary to use this type of job name syntax for the algorithm as the syntax used by JobScheduler does not recognize predecessor relationships (only successors).

Parallel nodes

We recommend that you use the following syntax for the names of job nodes that are processed in parallel:
"splitter job node name" ":" "job name".

In the example described above, one of the first nodes would then be split_partitions:export_table_partition_1.
This allows the diagram algorithmus in JOE to know and correctly display the nodes that directly follow on from the splitter. This is because the JobScheduler syntax does not recognise predecessor relationships (only successors).

Job nodes

As far as possible, the names of job nodes should identical to the job names (poss. without the folder name). If a job is used more than once in a job chain, then the node name can be uniquely specified using a letter or number as a suffix.

Error nodes

The name of the error node should either contain the job name or be identical with it. This means that in the event of an processing error in the job chain, it is possible to see immediately in JOC the point in the job chain where the abnormal termination occured.
In addition, the name should start with an "!" (an exclamation point, or with another unique special character). This makes it easier to see in the order history in JOC that the job chain has terminated abnormally.

See also:

Downloads

You can download the example described in this FAQ : insert_to_export_table_parallel.zip.

  • No labels