Page History

...

• Use Case
  • A number of orders are added in an arbitrary sequence and at arbitrary points in ime to a job chain - either from a file order source watching incoming files, from Ad Hoc orders or from permanent orders. The orders should be forced into serialized processing based on some predefined criteria.
  • Each order comes
    • either
    • with a parameter that specifies a numeric sequence,
    • or with a parameter that specifies a business date, e.g. for data files that are referenced by the order and that should be loaded to a data warehouse in sequence of the business dates,
    • or without parameters and should be processed in alphabetical or numeric sequence of its order id.
  • Orders should be processed strictly in sequence - either ascending or descending. If the next order does not provide the expected value then it is suspended and the job chain waits for an order with the expected value to arrive. After arrival of an order with the expected value all suspended orders are checked if they provide the expected value required for the successor order. 
• Solution Outline
  • A single job is added to the top of a job chain and that will
    • check for exepected values and suspend incoming orders until an order with the expected value arrives.
    • move orders that match expected values to the next job node in the job chain and restart evaluation of recheck the expected value of any suspended orders.
• References
  • SourceForge Forum Discussion Thread

...

• The solution implements a job expect that can be added to the top of any job chain.
  • This job implements a spooler_process() function that checks if an order matches the expected value and that suspends non-matching orders.
    • Expected values can be provided by incoming orders
      • by use of parameters:
        • each incoming order is assumed to provide a parameter that contains the expected value. 
          • Example for use with oder parameter: business_date = 2015-11-01
        • the name of the incoming orders' parameter is specified as the value of the control_order_expected_parameter parameter to the expect job.
          • Example for use with expect job parameter: control_order_expected_parameter= business_date
      • without parameters: 
        • the job will expect a numeric order id to be provided and increments the order id to calculcate the next expected value.
    • For expected values a function has to be provided that calculates the next expected value:
      • the function is specified by use of the conctrol_order_expected_value_function parameter to the expect job. The function is implemented with some JavaScript code that returns the next expected value based on the current value that is provided with the currentValue variable:
        • Example for numeric calculation: 
          • parseInt(currentValue) + 1
          • Explanation: the function parses the current expected numeric value and returns the incremented value
        • Example for date calculcation:
          • (new Date( (new Date(currentValue)).setDate( (new Date(currentValue)).getDate()+1 ) )).toISOString().substring(0,10);
          • Explanation: the function accepts the current date value, adds one day and returns the date in ISO format, e.g. 2015-11-01
    • For expected values a default value can be provided that is used to check the expected value of the first incoming order:
      • the default value can be specified by use of the control_order_expected_default_value parameter to the expect job.
        • Example for use with numeric default value: control_order_expected_default_value = 1
        • Example for use with date default value: control_order_expected_default_value = 2015-11-01
      • without specification of a default value the value of the first order processed by the expect job will be used as a default.
  • This job is configured for a single task, i.e. it executes incoming orders sequentually.
  • Having received an order with the expected value this job moves that order to the next job node in the job chain and activates any suspended orders to be re-checked to rechecked if they provide the next expected value.
• The solution automatically creates a control order that is used to store the next expected value.
  • The name of the control order is created from the prefix "control_order_" and the job node state that the expect job node is assigned.
  • The control order is suspended and not processed by subsequent job nodes. Its only purpose is to carry the next expected value for use of the solution with JobScheduler Agents and cluster members.
• The sample makes use of a job chain job_chain_expected_orders that includes the job nodes for the expect job and a hello job. The job chain accepts Ad Hoc orders that are added by use of JOC and the job chain can easily be modified to watch for incoming files and to create one order for each file.

...

• Add thre orders to the job_chain_expect_orders job chain. For your convenience the orders 1, 2 and 3 are provided with the sample.
  • Each order uses a parameter sequence with a sequential number such as 1, 2, 3.
  • The expect job uses the parameters 
    • control_order_expected_parameter with the value: sequence
    • control_order_expected_default_value with the value: 1
    • control_order_expected_value_function with the value: parseInt(currentValue) + 1
• Start order 1 that matches the expected value (parameter sequence=1). 
  • The order is processed and moved to the next_job job.
  • A control order control_order_expect is dynamically created and is suspended that contains the next expected value (parameter control_order_expect=2)
• Start order 3 that does not match the expected value (parameter sequence=3).
  • The order is suspended.
  • The control order with the expected value (parameter control_order_expect=2) remains unchanged.
• Start order 2 that does match the expected value (parameter sequence=2) .
  • Order 2 is processed and moved to the next_job job. 
    • The control order is automatically modified to carry the next expected value (parameter control_order_expect=3).
  • Order 3 is processed and moved to the next_job job.
    • The control order is automatically modified to carry the next expected value (parameter control_order_expect=4).
• For use with a date sequence instead of a numeric sequence 
  • the orders 1, 2, 3 carry a second parameter business_date with values in an ISO date format. 
  • the expect job carries sample parameters (prefixed with sample_) that can be activated (renamed) to work with the business_date parameter instead of the sequence parameter of the incoming orders .

...