How to prevent parallel execution of orders in job chains

Scope

• Use Case:
  • Consider the situation where a number of orders that have been added to a job chain. These orders should then be serialized to guarantee that each order has completed the job chain before the next order starts.
  • Consider the situation that a number of job chains makes use of the same resource, e.g. by access to objects in a database. These orders should be serialized to guarantee that only one order of one job chain can access the resource at the same time.
• Solution Outline:
  • The solution is twofold:
    • Resource Locks implement a job chain that accepts shadow orders for access to resources. The job chain guarantees that only one order at a time is granted a resource lock. The job chain will suspend orders in originating job chains as long as a resource lock is blocked and will continue such orders if the resource lock is available. Any number of orders of the same or of different job chains can use the Resource Locks job chain for serialization.
    • Resource Lock Consumers implements a sample job chain that makes use of a resource that may be accessed exclusively by one order at a time.
• References
  • SourceForge Forum Discussion Thread

Solution

• Download resource_locks.zip
• Download resource_lock_consumer.zip
• Extract the archives to the ./config/live folder of your JobScheduler installation.
• The archive extracts the files to the folders  resource_locks and resource_lock_consumer respectively. 
• You can store the resource_lock_consumer files in any folder as you like, however, if you move the resource_lock files to some other location then you will have to adjust settings in the resource_lock_consumer objects.

Pattern

Implementation

Components

• The solution implements a job named sorter that can be added at the start of any job chain.
  • This job implements a spooler_process() function that suspends all incoming orders.
  • This job is configured for a single task and with an idle timeout attribute. This means that it will execute incoming orders sequentially.
  • Having received the last available order this job will wait for the duration specified with the idle_timeout attribute for new orders. 
    • The idle timeout is configured using, for example <job idle_timeout="10"> with the sorter job definition.
    • Once the idle timeout has expired this job will execute its spooler_exit() function and then sort and move all orders that have previously been suspended.
      • Sorting is done in alphabetical order.
      • The orders are moved to the next job chain node that follows the sorter job in the job chain.
• The download example uses a job chain named job_chain1 that includes the job nodes for the sorter job and a hello job. This job chain accepts ad hoc orders that are added by JOC and it can easily be modified to watch for incoming files and to create an order for each file.
• Hint: to re-use the sorter job you can:
  • store the job in a central folder and reference the job in individual job chains.
  • move the job's JavaScript code to a central location and use an appropriate <include> element for individual job scripts.

function spooler_process_before() {
    var	resourceJobChainParamDefault = "/resource_locks/resource_locks";
    var resourceLockJobChainParam = "resource_job_chain";
    var resourceLockNameParam = "resource_lock";
    var resourceLockWakeUpParam = "resource_lock_wake_up";

	var order = spooler_task.order;
    var params = spooler.create_variable_set();
  		params.merge( spooler_task.params );
  		params.merge( order.params );

    // name of the requested resource: explicitely set by parameter or assuming the current job chain path
    var resourceLock = params.value( resourceLockNameParam );
    if ( !resourceLock ) {
		resourceLock = order.job_chain.path;
	}

    // name of the job chain that handles resource locks
    var resourceJobChainName = params.value( resourceLockJobChainParam );
    if ( !resourceJobChainName ) {
    	resourceJobChainName = resourceJobChainParamDefault;
    }

	// order wake up parameter indicates that order has acquired resource lock
    if ( params.value( resourceLockWakeUpParam ) == "yes" ) {
		spooler_log.info( ".. order acquired resource lock: " + resourceLock );
		return true;
    } else if ( params.value( resourceLockWakeUpParam ) == "no" ) {
		spooler_log.info( ".. order suspended and waiting for resource lock: " + resourceLock );
    	order.state = order.job_chain_node.state;
		order.suspended = true;
		return false;
	}

    var resourceJobChain = spooler.job_chain( resourceJobChainName );
    var resourceOrder = spooler.create_order();

    var resourceParams = spooler.create_variable_set();
    resourceParams.set_var( "requestor_job_chain", order.job_chain.path );
    resourceParams.set_var( "requestor_order", order.id );
    resourceOrder.params = resourceParams;
	resourceOrder.title = resourceLock;
	resourceOrder.at = "now+5";

    if ( (order.job_chain.path + "/" + order.id).length <= 100 ) {
		resourceOrder.id = order.job_chain.path + "/" + order.id;
    } 

    resourceJobChain.add_order( resourceOrder );
    spooler_log.info( ".. order has requested resource lock: " + resourceLock );

	params.set_var( resourceLockWakeUpParam, "no" );
    order.state = order.job_chain_node.state;

    // always suspend current order, it will be waked up by the requested resource lock
    order.suspended = true;

	return false;
}

Usage

• Add two orders to the job_chain1 job chain. 
  • Use an order ID in descending alphabetical order, e.g. "cba" for the order ID of the first order and "abc" for the order ID of the second.
• Both orders will be suspended at the first node of the job chain.
• After an idle timeout of 10s both orders will be moved to the next job node in the job chain. 
  • This time the orders will be processed in ascending alphabetical order.