Page History

Table of Contents

Scope

Use Cases:
- Consider the situation where a number of orders that have been added to a job chain. These orders should then be serialized to guarantee that each order has completed the job chain before the next order starts.
- Consider the situation that a number of job chains makes use of the same resource, e.g. by access to the same objects in a database. These orders should be serialized to guarantee that only one order of one job chain can access the resource at the same time.
Requirements
- The solution implements resource locks that will serialize access
  - for orders of any number of job chains.
  - for any number of orders of the same job chain
- Resource locks are persistent, i.e. they are restored to the same status after a JobScheduler restart.
- Resource locks can by acquired by job chains that are executed
  - with JobScheduler Agent instances.
  - with JobScheduler Master instances running standalone or in a Passive Cluster or Active Cluster.
- Resource locks can be monitored with the JOC GUI.
Delimitation
- The solution is focused on exclusive locks, shared locks are not considered.
- Persistence of resource locks is limited to a predefined duration. The default value is 24 hrs. and can be configured for individual needs.
Solution Outline:
- The solution implements two roles represented by job chains:
  - Resource Lock Provider implements a job chain that accepts shadow orders for serialized access to resources. The job chain guarantees that only one order at a time is granted a resource lock. The job chain will suspend orders in originating job chains as long as a resource lock is blocked and will continue such orders when the resource lock becomes available. Any number of orders of the same or of different job chains can use the Resource Lock Provider job chain for serialization.
  - Resource Lock Consumer implements a sample job chain that makes use of a resource that should be accessed exclusively by one order at a time.
References
- SourceForge Forum Discussion Thread

Solution

Download resource_lock_provider.zip
Download resource_lock_consumer.zip
Extract the archives to the ./config/live folder of your JobScheduler installation.
The archive extracts the files to the folders resource_lock_provider and resource_lock_consumer respectively.
You can store the resource_lock_consumer files in any folder as you like, however, if you move the resource_lock_provider files to some other location then you will have to adjust settings in the resource_lock_consumer objects.

Pattern

Flowchart

chain_resource_lock [label="Job Chain\nimplements\na Resource Lock Provider",fillcolor="orange"]
job_manage_resource_lock [label="Job Manage Resource Lock",fillcolor="lightskyblue"]
resource_lock_available [shape=diamond,label="Resource Lock available?",fillcolor="white"]
 
chain_resource_lock_consumer_1 [label="Job Chain\nimplements a\nResource Lock Consumer",fillcolor="orange"]
job_request_resource_lock_1 [label="Job Request Resource Lock",fillcolor="lightskyblue"]
job_step_1_1 [label="Job Step 1",fillcolor="lightskyblue"]
job_step_1_2 [label="Job Step 2",fillcolor="lightskyblue"]

chain_resource_lock_consumer_2 [label="Job Chain\nimplements a\nResource Lock Consumer",fillcolor="orange"]
job_request_resource_lock_2 [label="Job Request Resource Lock",fillcolor="lightskyblue"]
job_step_2_1 [label="Job Step 1",fillcolor="lightskyblue"]
job_step_2_2 [label="Job Step 2",fillcolor="lightskyblue"]

order_1A [shape="ellipse",label="Order 1A",fillcolor="violet"]
order_1B [shape="ellipse",label="Order 1B",fillcolor="violet"]
order_2A [shape="ellipse",label="Order 2A",fillcolor="violet"]
order_2B [shape="ellipse",label="Order 2B",fillcolor="violet"]

shadow_order_1A [shape="ellipse",label="Shadow Order 1A",fillcolor="violet"]
shadow_order_1B [shape="ellipse",label="Shadow Order 1B",fillcolor="violet"]
shadow_order_2A [shape="ellipse",label="Shadow Order 2A",fillcolor="violet"]
shadow_order_2B [shape="ellipse",label="Shadow Order 2B",fillcolor="violet"]

order_suspend [label="Suspend Original Order",fillcolor="white"]
order_resume [label="Resume Original Order for Next Job",fillcolor="white"]

chain_resource_lock_consumer_1 -> order_1A
chain_resource_lock_consumer_1 -> order_1B
order_1A -> job_request_resource_lock_1 -> job_step_1_1
order_1B -> job_request_resource_lock_1 -> job_step_1_2
job_request_resource_lock_1 -> shadow_order_1A
job_request_resource_lock_1 -> shadow_order_1B
 
chain_resource_lock_consumer_2 -> order_2A
chain_resource_lock_consumer_2 -> order_2B
order_2A -> job_request_resource_lock_2 -> job_step_2_1
order_2B -> job_request_resource_lock_2 -> job_step_2_2
job_request_resource_lock_2 -> shadow_order_2A
job_request_resource_lock_2 -> shadow_order_2B
 
chain_resource_lock -> shadow_order_1A
chain_resource_lock -> shadow_order_1B
chain_resource_lock -> shadow_order_2A
chain_resource_lock -> shadow_order_2B
shadow_order_1A -> job_manage_resource_lock
shadow_order_1B -> job_manage_resource_lock
shadow_order_2A -> job_manage_resource_lock
shadow_order_2B -> job_manage_resource_lock

job_manage_resource_lock -> resource_lock_available
resource_lock_available -> order_resume [label="&nbsp;yes&nbsp;"]
resource_lock_available -> order_suspend [label="&nbsp;no&nbsp;"]
order_resume -> job_step_1_1 -> job_step_1_2
order_resume -> job_step_2_1 -> job_step_2_2

Implementation

Components

The solution implements a job named sorter that can be added at the start of any job chain.
- This job implements a spooler_process() function that suspends all incoming orders.
- This job is configured for a single task and with an idle timeout attribute. This means that it will execute incoming orders sequentially.
- Having received the last available order this job will wait for the duration specified with the idle_timeout attribute for new orders.
  - The idle timeout is configured using, for example <job idle_timeout="10"> with the sorter job definition.
  - Once the idle timeout has expired this job will execute its spooler_exit() function and then sort and move all orders that have previously been suspended.
    - Sorting is done in alphabetical order.
    - The orders are moved to the next job chain node that follows the sorter job in the job chain.
The download example uses a job chain named job_chain1 that includes the job nodes for the sorter job and a hello job. This job chain accepts ad hoc orders that are added by JOC and it can easily be modified to watch for incoming files and to create an order for each file.
Hint: to re-use the sorter job you can:
- store the job in a central folder and reference the job in individual job chains.
- move the job's JavaScript code to a central location and use an appropriate <include> element for individual job scripts.

...

Code Block

language	js
title	Monitor request_resource_lock
collapse	true

function spooler_process_before() {
    var	resourceJobChainParamDefault = "/resource_locks/resource_locks";
    var resourceLockJobChainParam = "resource_job_chain";
    var resourceLockNameParam = "resource_lock";
    var resourceLockWakeUpParam = "resource_lock_wake_up";

	var order = spooler_task.order;
    var params = spooler.create_variable_set();
  		params.merge( spooler_task.params );
  		params.merge( order.params );

    // name of the requested resource: explicitely set by parameter or assuming the current job chain path
    var resourceLock = params.value( resourceLockNameParam );
    if ( !resourceLock ) {
		resourceLock = order.job_chain.path;
	}

    // name of the job chain that handles resource locks
    var resourceJobChainName = params.value( resourceLockJobChainParam );
    if ( !resourceJobChainName ) {
    	resourceJobChainName = resourceJobChainParamDefault;
    }

	// order wake up parameter indicates that order has acquired resource lock
    if ( params.value( resourceLockWakeUpParam ) == "yes" ) {
		spooler_log.info( ".. order acquired resource lock: " + resourceLock );
		return true;
    } else if ( params.value( resourceLockWakeUpParam ) == "no" ) {
		spooler_log.info( ".. order suspended and waiting for resource lock: " + resourceLock );
    	order.state = order.job_chain_node.state;
		order.suspended = true;
		return false;
	}

    var resourceJobChain = spooler.job_chain( resourceJobChainName );
    var resourceOrder = spooler.create_order();

    var resourceParams = spooler.create_variable_set();
    resourceParams.set_var( "requestor_job_chain", order.job_chain.path );
    resourceParams.set_var( "requestor_order", order.id );
    resourceOrder.params = resourceParams;
	resourceOrder.title = resourceLock;
	resourceOrder.at = "now+5";

    if ( (order.job_chain.path + "/" + order.id).length <= 100 ) {
		resourceOrder.id = order.job_chain.path + "/" + order.id;
    } 

    resourceJobChain.add_order( resourceOrder );
    spooler_log.info( ".. order has requested resource lock: " + resourceLock );

	params.set_var( resourceLockWakeUpParam, "no" );
    order.state = order.job_chain_node.state;

    // always suspend current order, it will be waked up by the requested resource lock
    order.suspended = true;

	return false;
}

Usage

Add two orders to the job_chain1 job chain.
- Use an order ID in descending alphabetical order, e.g. "cba" for the order ID of the first order and "abc" for the order ID of the second.
Both orders will be suspended at the first node of the job chain.
After an idle timeout of 10s both orders will be moved to the next job node in the job chain.
- This time the orders will be processed in ascending alphabetical order.

...

Space shortcuts

Page tree

Versions Compared

Old Version 28

New Version 29

Key

Scope

Solution

Pattern

Implementation

Components

Usage