Page History

...

Solution

• Use CaseCases:
  • Consider the situation where a number of orders that have been added to a job chain. These orders should then be serialized to guarantee that each order has completed the job chain before the next order starts.
  • Consider the situation that a number of job chains makes use of the same resource, e.g. by access to the same objects in a database. These orders should be serialized to guarantee that only one order of one job chain can access the resource at the same time.
• Solution Outline:
  • The solution is twofold:
    • Resource Locks implement a job chain that accepts shadow orders for access to resources. The job chain guarantees that only one order at a time is granted a resource lock. The job chain will suspend orders in originating job chains as long as a resource lock is blocked and will continue such orders if the resource lock is available. Any number of orders of the same or of different job chains can use the Resource Locks job chain for serialization.
    • Resource Lock Consumers implements a sample job chain that makes use of a resource that may be accessed exclusively by one order at a time.
• References
  • SourceForge Forum Discussion Thread

Solution

• Download resource_locks.zip
• Download resource_lock_consumer.zip
• Extract the archives to the ./config/live folder of your JobScheduler installation.
• The archive extracts the files to the folders  resource_locks and resource_lock_consumer respectively. 
• You can store the resource_lock_consumer files in any folder as you like, however, if you move the resource_lock files to some other location then you will have to adjust settings in the resource_lock_consumer objects.

Pattern

• • This use case applies e.g. to NoSQL databases that provide limited data consistency and therefore require consistency to be provided at access level by a JobScheduler.
• Requirements
  • The solution implements resource locks that serialize access 
    • for orders of any number of job chains,
    • for any number of orders of the same job chain,
    • for orders that are added to any job node in a job chain.
  • Resource locks are persistent, i.e. they are restored to the same state after a JobScheduler restart.
  • Resource locks can by acquired by job chains that are executed 
    • with JobScheduler Agent instances including multiple Agents for job nodes of the same job chain (cross-platform).
    • with JobScheduler Master instances running standalone or in a Passive Cluster or Active Cluster.
  • Resource locks can be monitored and can manually be removed with the JOC GUI should the need arise.
• Delimitation
  • Limiting the number of concurrent orders in a job chain by use of the <job_chain max_orders="..."/> attribute works exclusively for the first job node. The solution offered by this article works optionally on any job node in a job chain.
  • This solution is focused on 
    • locks for complete job chains. For locks at job (task) level see Locks.
    • exclusive locks, shared locks are not considered.
  • Persistence of resource locks is limited to a predefined duration. The default value is 24 hrs. and can be configured for individual needs.
• Solution Outline:
  • The solution implements two parts represented by job chains:
    • Resource Lock Provider implements a job chain that accepts shadow orders for serialized access to resources. The job chain guarantees that only one order at a time is granted a resource lock. The job chain will suspend orders in originating job chains as long as a resource lock is blocked and will continue such orders when the resource lock becomes available. Any number of orders of the same or of different job chains can use the Resource Lock Provider job chain for serialization.
    • Resource Lock Consumer implements a sample job chain that makes use of a resource that should be accessed exclusively by one order at a time.
• References
  • SourceForge Forum Discussion Thread

  • Jira
    server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JS-402

Download

• Download resource_lock_provider.zip
• Download resource_lock_consumer.zip
• Extract the archives to the ./config/live folder of your JobScheduler installation.
• The archive extracts the files to the folders  resource_lock_provider and resource_lock_consumer respectively. 
• You can store the resource_lock_consumer files in any folder as you like, however, if you move the resource_lock_provider files to some other location then you will have to adjust settings in the resource_lock_consumer objects.

Pattern

chain_resource_lock [label="Job Chain\nimplements\na Resource Lock Provider",fillcolor="orange"]
job_manage_resource_lock [label="Job Manage Resource Lock

chain_resource_lock [label="Job Chain\nrepresents\na resource manager",fillcolor="orange"]
job_manage_resource_lock [label="Job Manage Resource Lock",fillcolor="lightskyblue"]
resource_lock_available [shape=diamond,label="resourece lock available?",fillcolor="white"]
 
chain_resource_lock_consumer_1 [label="Job Chain\nrepresents a\nresource lock consumer",fillcolor="orange"]
job_request_resource_lock_1 [label="Job Request Resource Lock",fillcolor="lightskyblue"]
job_step_1_1 [label="Job Step 1",fillcolor="lightskyblue"]
job_step_1_2 [label="Job Step 2",fillcolor="lightskyblue"]

chain_resource_lock_consumer_2 [label="Job Chain\ntrepresents a\nresource lock consumer",fillcolor="orange"]
job_request_resource_lock_2 [label="Job Request Resource Lock",fillcolor="lightskyblue"]
job_step_2_1 [label="Job Step 1",fillcolor="lightskyblue"]
jobresource_steplock_2_2available [shape=diamond,label="JobResource StepLock 2available?",fillcolor="lightskybluewhite"]

order_1A [shape="ellipse", 
chain_resource_lock_consumer_1 [label="Order 1AJob Chain\nimplements a\nResource Lock Consumer",fillcolor="violetorange"]
order_1B [shape="ellipse",job_request_resource_lock_1 [label="Order 1BJob Request Resource Lock",fillcolor="violetlightskyblue"]
order_2Ajob_step_1_1 [shapelabel="ellipseJob Step 1",labelfillcolor="Order 2A",fillcolor="violet"]
order_2B [shape="ellipse",label="Order 2Blightskyblue"]
job_step_1_2 [label="Job Step 2",fillcolor="violetlightskyblue"]

shadow_order_1A [shape="ellipse",chain_resource_lock_consumer_2 [label="ShadowJob Order 1A"Chain\nimplements a\nResource Lock Consumer",fillcolor="violetorange"]
shadow_order_1B [shape="ellipse",job_request_resource_lock_2 [label="ShadowJob Request OrderResource 1BLock",fillcolor="violetlightskyblue"]
shadowjob_step_order2_2A1 [shape="ellipse",label="ShadowJob OrderStep 2A1",fillcolor="violetlightskyblue"]
shadowjob_step_order2_2B2 [shape="ellipse",label="ShadowJob OrderStep 2B2",fillcolor="violetlightskyblue"]

order_suspend_11A [shape="ellipse",label="SuspendOrder Order1A",fillcolor="whiteviolet"]
order_wait_11B [shape="ellipse",label="Wait for next OrderOrder 1B",fillcolor="whiteviolet"]
order_move_12A [shape="ellipse",label="Move Orders to Next JobOrder 2A",fillcolor="violet"]
order_2B [shape="ellipse",label="Order 2B",fillcolor="whiteviolet"]

chain_resource_lock_consumer_1 -> order_1A
chain_resource_lock_consumer_1 -> order_1B
order_1A -> job_request_resource_lock_1
order_1B -> job_request_resource_lock_1
job_request_resource_lock_1 -> job_manage_resource_lock

 
#resource_lock_available_1 -> order_move_1 [label=" yes "]
#resource_lock_availableshadow_order_1A [shape="ellipse",label="Shadow Order 1A",fillcolor="violet"]
shadow_order_1B [shape="ellipse",label="Shadow Order 1B",fillcolor="violet"]
shadow_order_2A [shape="ellipse",label="Shadow Order 2A",fillcolor="violet"]
shadow_order_2B [shape="ellipse",label="Shadow Order 2B",fillcolor="violet"]

order_suspend [label="Suspend Original Order",fillcolor="white"]
order_resume [label="Resume Original Order for Next Job",fillcolor="white"]

chain_resource_lock_consumer_1 -> order_suspend_1 [label=" no "]
#order_move_11A
chain_resource_lock_consumer_1 -> order_1B
order_1A -> job_request_stepresource_1lock_1 -> job_step_1_21
 
chain_resource_lockorder_1B -> shadow> job_orderrequest_1A
chain_resource_lock_1 -> shadowjob_step_order1_1B
shadow_order_1A -> 2
job_managerequest_resource_lock
shadow_order_1B1 -> jobshadow_manage_resource_lock

order_1A
job_managerequest_resource_lock_1 -> resourceshadow_lockorder_available
job_manage1B
 
chain_resource_lock_consumer_2 -> order_2A
chain_resource_lock_available



 

Implementation

Components

• The solution implements a job named sorter that can be added at the start of any job chain.
  • This job implements a spooler_process() function that suspends all incoming orders.
  • This job is configured for a single task and with an idle timeout attribute. This means that it will execute incoming orders sequentially.
  • Having received the last available order this job will wait for the duration specified with the idle_timeout attribute for new orders. 
    • The idle timeout is configured using, for example <job idle_timeout="10"> with the sorter job definition.
    • Once the idle timeout has expired this job will execute its spooler_exit() function and then sort and move all orders that have previously been suspended.
      • Sorting is done in alphabetical order.
      • The orders are moved to the next job chain node that follows the sorter job in the job chain.
• The download example uses a job chain named job_chain1 that includes the job nodes for the sorter job and a hello job. This job chain accepts ad hoc orders that are added by JOC and it can easily be modified to watch for incoming files and to create an order for each file.
• Hint: to re-use the sorter job you can:
  • store the job in a central folder and reference the job in individual job chains.
  • move the job's JavaScript code to a central location and use an appropriate <include> element for individual job scripts.

function spooler_process_before() {
    var	resourceJobChainParamDefault = "/resource_locks/resource_locks";
    var resourceLockJobChainParam = "resource_job_chain";
    var resourceLockNameParam = "resource_lock";
    var resourceLockWakeUpParam = "resource_lock_wake_up";

	var order = spooler_task.order;
    var params = spooler.create_variable_set();
  		params.merge( spooler_task.params );
  		params.merge( order.params );

    // name of the requested resource: explicitely set by parameter or assuming the current job chain path
    var resourceLock = params.value( resourceLockNameParam );
    if ( !resourceLock ) {
		resourceLock = order.job_chain.path;
	}

    // name of the job chain that handles resource locks
    var resourceJobChainName = params.value( resourceLockJobChainParam );
    if ( !resourceJobChainName ) {
    	resourceJobChainName = resourceJobChainParamDefault;
    }

	// order wake up parameter indicates that order has acquired resource lock
    if ( params.value( resourceLockWakeUpParam ) == "yes" ) {
		spooler_log.info( ".. order acquired resource lock: " + resourceLock );
		return true;
    } else if ( params.value( resourceLockWakeUpParam ) == "no" ) {
		spooler_log.info( ".. order suspended and waiting for resource lock: " + resourceLock );
    	order.state = order.job_chain_node.state;
		order.suspended = true;
		return false;
	}

    var resourceJobChain = spooler.job_chain( resourceJobChainName );
    var resourceOrder = spooler.create_order();

    var resourceParams = spooler.create_variable_set();
    resourceParams.set_var( "requestor_job_chain", order.job_chain.path );
    resourceParams.set_var( "requestor_order", order.id );
    resourceOrder.params = resourceParams;
	resourceOrder.title = resourceLock;
	resourceOrder.at = "now+5";

    if ( (order.job_chain.path + "/" + order.id).length <= 100 ) {
		resourceOrder.id = order.job_chain.path + "/" + order.id;
    } 

    resourceJobChain.add_order( resourceOrder );
    spooler_log.info( ".. order has requested resource lock: " + resourceLock );

	params.set_var( resourceLockWakeUpParam, "no" );
    order.state = order.job_chain_node.state;

    // always suspend current order, it will be waked up by the requested resource lock
    order.suspended = true;

	return false;
}

Usage

_consumer_2 -> order_2B
order_2A -> job_request_resource_lock_2 -> job_step_2_1
order_2B -> job_request_resource_lock_2 -> job_step_2_2
job_request_resource_lock_2 -> shadow_order_2A
job_request_resource_lock_2 -> shadow_order_2B
 
chain_resource_lock -> shadow_order_1A
chain_resource_lock -> shadow_order_1B
chain_resource_lock -> shadow_order_2A
chain_resource_lock -> shadow_order_2B
shadow_order_1A -> job_manage_resource_lock
shadow_order_1B -> job_manage_resource_lock
shadow_order_2A -> job_manage_resource_lock
shadow_order_2B -> job_manage_resource_lock

job_manage_resource_lock -> resource_lock_available
resource_lock_available -> order_resume [label=" yes "]
resource_lock_available -> order_suspend [label=" no "]
order_resume -> job_step_1_1 -> job_step_1_2
order_resume -> job_step_2_1 -> job_step_2_2

Implementation

Job: request_resource_lock

• This job can be used by resource lock consumers as the first job node in a job chain.
• The job has no special job script implementation.
  
  Image Added
  
  

• The job makes use of the Monitor request_resource_lock.
  
  Image Added

Monitor: request_resource_lock

• This Monitor implements a pre-processing script for jobs that request resource locks.
• The monitor is used by the above request_resource_lock job and can be assigned to any individual job node of a job chain that requires a resource lock.

function spooler_process_before() {
 
    var	resourceJobChainParamDefault = "/resource_lock_provider/resource_lock_provider";
    var resourceLockJobChainParam = "resource_job_chain";
    var resourceLockNameParam = "resource_lock";
    var resourceLockWakeUpParam = "resource_lock_wake_up";

	var order = spooler_task.order;
    var params = spooler.create_variable_set();
  	params.merge( spooler_task.params );
  	params.merge( order.params );

    // name of the requested resource: explicitely set by parameter or assuming the current job chain path
    var resourceLock = params.value( resourceLockNameParam );
    if ( !resourceLock ) {
		resourceLock = order.job_chain.path;
	}

    // name of the job chain that handles resource locks
    var resourceJobChainName = params.value( resourceLockJobChainParam );
    if ( !resourceJobChainName ) {
    	resourceJobChainName = resourceJobChainParamDefault;
    }

	// order wake up parameter indicates that order has acquired resource lock
    if ( params.value( resourceLockWakeUpParam ) == "yes" ) {
		spooler_log.info( ".. order acquired resource lock: " + resourceLock );
		return true;
    } else if ( params.value( resourceLockWakeUpParam ) == "no" ) {
		spooler_log.info( ".. order suspended and waiting for resource lock: " + resourceLock );
    	order.state = order.job_chain_node.state;
		order.suspended = true;
		return false;
	}

    var resourceJobChain = spooler.job_chain( resourceJobChainName );
    var resourceOrder = spooler.create_order();

    var resourceParams = spooler.create_variable_set();
    resourceParams.set_var( "requestor_job_chain", order.job_chain.path );
    resourceParams.set_var( "requestor_order", order.id );
    resourceOrder.params = resourceParams;

	resourceOrder.title = resourceLock;
	resourceOrder.at = "now+2";

    if ( (order.job_chain.path + "/" + order.id).length <= 100 ) {
		resourceOrder.id = order.job_chain.path + "/" + order.id;
    } 

    resourceJobChain.add_order( resourceOrder );
    spooler_log.info( ".. order requested resource lock: " + resourceLock );

	params.set_var( resourceLockWakeUpParam, "no" );
    order.state = order.job_chain_node.state;

    // always suspend current order, it will be waked up by the requested resource lock
    order.suspended = true;

	return false;
}

Job: manage_resource_lock

• This job provides the handling of resource locks.
  • For incoming shadow orders that have been created by the request_resource_lock monitor the originating orders are resumed if the resource lock is not blocked by a different order..
  • Shadow orders are repeatedly checked by this job by use of a setback interval.
  • Shadow orders are completed if the originating order has completed the job chain or has been removed.

Configuration

• Use the sample job chain test_case_1 from the Resource Lock Consumer sample or any individual job chain and assign the job /resource_lock_provider/request_resource_lock as the first job node of the job chain. This configuration will guarantee that all newly created orders that are added to the begin of the job chain will be started after any predecessor orders completed the job chain.
  
  Image Added 

• Should resource lock consistency be required for later job nodes in a job chain, i.e. if orders are created not for the begin of a job chain but for later job nodes, e.g. when using split job nodes, then assign the monitor /resource_lock_provider/request_resource_lock to the respective jobs:
  
  Image Added 
   
• The repeat interval for which resource locks are checked is configured with the job manage_resource_lock of the resource lock provider.
  • The interval is 60s per default
  • The maximum number of retries is 1440 which defaults to a lifetime of 24 hrs. for a resource lock.
• The configuration is provided like this:
  
  Image Added 

Hints

• When requesting a resource lock by use of the request_resource_lock monitor then 
  • a delay of 2s is applied for the manage_resource_lock job to process the shadow order. A minimum delay is required to guarantee synchronization.
  • for shell jobs an error message will be added to the order log when the order is initially suspended. This message is not harmful and can be ignored. For API jobs no errors or warnings are raised. We recommend that you use the request_resource_lock job as a first job node of your job chain and use the monitor for subsequent job nodes only.
• The lifetime of a resource lock is limited by the duration that is covered with the setback interval of the manage_resource_lock job that defaults to 24 hrs.
• If you make frequent use of resource locks then you should consider to pre-load the manage_resource_lock job by use of the setting <job min_tasks="1"/> which results in the job to react immediately to incoming orders without having to load a JVM for each execution of this job.
• When operating the Resource Lock Provider job chain in a different location than the directory ./config/live/resource_lock_provider then the following settings have to be adjusted:
  • From the Resource Lock Provider the request_resource_lock monitor script specifies the path of the provider job chain.
  • From the samples for the Resource Lock Consumer
    • the job chains test_case_1 and test_case_1A both assign the job /resource_lock_provider/request_resource_lock as the first job node.
    • the job chain test_case_2 makes use of jthe obs job_step_1_resource_lock and job_step_2_resource_lock that are assigned the monitor location in /resource_lock_provider/request_resource_lock.

Usage

The following test cases are available with the Resource Lock Consumer sample job chain.

Test case 1: Check resource lock on first job step

• • The job chain test_case_1 makes use of the /resource_lock_provider/request_resource_lock job as the first job node.
  • No parameterization is used, therefore the scope of the resource lock is the current job chain that is forced to process order sequentially.
  • Adding multiple orders to this job chain causes all orders to be processed only after predecessor orders completed the job chain.
    • Start the orders test_case_1-1, ..., test_case_1-3 by use of JOC.
    • Add any number of new orders by use of JOC with the Add Order context menu.

Test case 1A: Lock job chains

• • The job chain test_case_1A is a copy of test_case_1 but makes use of the same resource lock as test_case_1. 
  • The orders test_case_1A-1, ..., test_case_1A-3 use the parameter resource_lock with the value /resource_lock_consumer/test_case_1 therefore forcing orders for both job chains to be processed sequentially:
    • Start the orders test_case_1-1, ..., test_case_1-3 by use of JOC that will run for job chain test_case_1.
    • Start the orders test_case_1A-1, ..., test_case_1A-3 by use of JOC that will run in job chain test_case_1A but are synchronized with orders of job chain test_case_1.

Test case 2: Check resource lock on each job step

• • This test case is designed to prevent parallel execution of orders in a job chain independent from the job node that orders are started for.
  • The job chain test_case_2 makes use of the jobs job_step_1_resource_lock and job_step_2_resource_lock. Both jobs are assigned the request_resource_lock monitor. The above-mentioned initial job node request_resource_lock is not used as the individual jobs make use of the resource lock monitor.
  • Adding multiple orders to this job chain causes orders to behave sequentially as for test case 1:
    
    • Start the orders test_case_2-1, ..., test_case_2-3 by use of JOC.
    • Add any number of new orders by use of JOC with the Add Order context menu
• Add two orders to the job_chain1 job chain. 
  • Use an order ID in descending alphabetical order, e.g. "cba" for the order ID of the first order and "abc" for the order ID of the second.
• Both orders will be suspended at the first node of the job chain.
After an idle timeout of 10s both orders will be moved to the next job node in the job chain. This time the orders will be processed in ascending alphabetical order
• • • .