How to prevent concurrent access to resources by parallel execution of orders in job chains

Scope

• Use Cases:
  • Consider the situation where a number of orders have been added to a job chain. These orders should then be serialized to guarantee that each order has completed the job chain before the next order starts.
  • Consider the situation that a number of job chains makes use of the same resource, e.g. by access to the same objects in a database. These orders should be serialized to guarantee that only one order of one job chain can access the resource at the same time. This use case applies e.g. to NoSQL databases that provide limited data consistency and therfore require consistency to be provided at access level by a JobScheduler.
• Requirements
  • The solution implements resource locks that serialize access 
    • for orders of any number of job chains,
    • for any number of orders of the same job chain,
    • for orders that are added to any job node in a job chain.
  • Resource locks are persistent, i.e. they are restored to the same state after a JobScheduler restart.
  • Resource locks can by acquired by job chains that are executed 
    • with JobScheduler Agent instances.
    • with JobScheduler Master instances running standalone or in a Passive Cluster or Active Cluster.
  • Resource locks can be monitored and can manually be removed with the JOC GUI should the need arise.
• Delimitation
  • Limiting the number of concurrent orders in a job chain by use of the <job_chain max_orders="..."/> attribute works exclusively for the first job node. The solution works on any job node in a job chain.
  • The solution is focused on 
    • locks for complete job chains. For locks at job level see Locks.
    • exclusive locks, shared locks are not considered.
  • Persistence of resource locks is limited to a predefined duration. The default value is 24 hrs. and can be configured for individual needs.
• Solution Outline:
  • The solution implements two roles represented by job chains:
    • Resource Lock Provider implements a job chain that accepts shadow orders for serialized access to resources. The job chain guarantees that only one order at a time is granted a resource lock. The job chain will suspend orders in originating job chains as long as a resource lock is blocked and will continue such orders when the resource lock becomes available. Any number of orders of the same or of different job chains can use the Resource Lock Provider job chain for serialization.
    • Resource Lock Consumer implements a sample job chain that makes use of a resource that should be accessed exclusively by one order at a time.
• References
  • SourceForge Forum Discussion Thread
  • JS-402 - Getting issue details... STATUS

Solution

• Download resource_lock_provider.zip
• Download resource_lock_consumer.zip
• Extract the archives to the ./config/live folder of your JobScheduler installation.
• The archive extracts the files to the folders  resource_lock_provider and resource_lock_consumer respectively. 
• You can store the resource_lock_consumer files in any folder as you like, however, if you move the resource_lock_provider files to some other location then you will have to adjust settings in the resource_lock_consumer objects.

Pattern

Implementation

Components

• The solution implements a job named sorter that can be added at the start of any job chain.
  • This job implements a spooler_process() function that suspends all incoming orders.
  • This job is configured for a single task and with an idle timeout attribute. This means that it will execute incoming orders sequentially.
  • Having received the last available order this job will wait for the duration specified with the idle_timeout attribute for new orders. 
    • The idle timeout is configured using, for example <job idle_timeout="10"> with the sorter job definition.
    • Once the idle timeout has expired this job will execute its spooler_exit() function and then sort and move all orders that have previously been suspended.
      • Sorting is done in alphabetical order.
      • The orders are moved to the next job chain node that follows the sorter job in the job chain.
• The download example uses a job chain named job_chain1 that includes the job nodes for the sorter job and a hello job. This job chain accepts ad hoc orders that are added by JOC and it can easily be modified to watch for incoming files and to create an order for each file.
• Hint: to re-use the sorter job you can:
  • store the job in a central folder and reference the job in individual job chains.
  • move the job's JavaScript code to a central location and use an appropriate <include> element for individual job scripts.

Job: request_resource_lock

• This job can be used by resource lock consumers as the first job node in a job chain.
• The job has no special job script implementation.
  
  
  
  

• The job makes use of the Monitor request_resource_lock.
  
  

Monitor: request_resource_lock

• This Monitor implements a pre-processing script for jobs that request resource locks.
• The monitor is used by the above request_resource_lock job and can be assigned to any individual job node of a job chain that requires a resource lock.

function spooler_process_before() {
 
    var	resourceJobChainParamDefault = "/resource_lock_provider/resource_lock_provider";
    var resourceLockJobChainParam = "resource_job_chain";
    var resourceLockNameParam = "resource_lock";
    var resourceLockWakeUpParam = "resource_lock_wake_up";

	var order = spooler_task.order;
    var params = spooler.create_variable_set();
  		params.merge( spooler_task.params );
  		params.merge( order.params );

    // name of the requested resource: explicitely set by parameter or assuming the current job chain path
    var resourceLock = params.value( resourceLockNameParam );
    if ( !resourceLock ) {
		resourceLock = order.job_chain.path;
	}

    // name of the job chain that handles resource locks
    var resourceJobChainName = params.value( resourceLockJobChainParam );
    if ( !resourceJobChainName ) {
    	resourceJobChainName = resourceJobChainParamDefault;
    }

	// order wake up parameter indicates that order has acquired resource lock
    if ( params.value( resourceLockWakeUpParam ) == "yes" ) {
		spooler_log.info( ".. order acquired resource lock: " + resourceLock );
		return true;
    } else if ( params.value( resourceLockWakeUpParam ) == "no" ) {
		spooler_log.info( ".. order suspended and waiting for resource lock: " + resourceLock );
    	order.state = order.job_chain_node.state;
		order.suspended = true;
		return false;
	}

    var resourceJobChain = spooler.job_chain( resourceJobChainName );
    var resourceOrder = spooler.create_order();

    var resourceParams = spooler.create_variable_set();
    resourceParams.set_var( "requestor_job_chain", order.job_chain.path );
    resourceParams.set_var( "requestor_order", order.id );
    resourceOrder.params = resourceParams;

	resourceOrder.title = resourceLock;
	resourceOrder.at = "now+3";

    if ( (order.job_chain.path + "/" + order.id).length <= 100 ) {
		resourceOrder.id = order.job_chain.path + "/" + order.id;
    } 

    resourceJobChain.add_order( resourceOrder );
    spooler_log.info( ".. order requested resource lock: " + resourceLock );

	params.set_var( resourceLockWakeUpParam, "no" );
    order.state = order.job_chain_node.state;

    // always suspend current order, it will be waked up by the requested resource lock
    order.suspended = true;

	return false;
}

Configuration

• Use the sample job chain test_case_1 from the Resource Lock Consumer sample or any individual job chain and assign the job /resource_lock_provider/request_resource_lock as the first job node of the job chain. This configuration will guarantee that all newly created orders that are added to the begin of the job chain will be started after any predecessor orders completed the job chain.
  
   

• Should resource lock consistency be required for later job nodes in a job chain, i.e. if orders are created not for the begin of a job chain but for later job nodes, e.g. when using split job nodes, then assign the monitor /resource_lock_provider/request_resource_lock to the respective jobs:
  
   

Usage

• Add two orders to the job_chain1 job chain. 
  • Use an order ID in descending alphabetical order, e.g. "cba" for the order ID of the first order and "abc" for the order ID of the second.
• Both orders will be suspended at the first node of the job chain.
• After an idle timeout of 10s both orders will be moved to the next job node in the job chain. 
  • This time the orders will be processed in ascending alphabetical order.