Page History

...

• Use Case:
  • Consider the situation where a number of orders are added in an arbitrary sequence and at arbitrary points in time to a job chain - either from a file order source watching incoming files, from ad hoc orders or from permanent orders. The orders should then be forced into serialized processing processed serially according to some predefined criteria.
  • Each order comes either:
    
    • with a parameter that specifies a numeric sequence,
    • with a parameter that specifies a business date, e.g. for data files that are referenced by the order and that should be loaded to a data warehouse in sequence of the business dates,
    • without parameters and should be processed in alphabetical or numerical sequence according to its order ID.
  • Orders should be processed strictly in the required sequence - either ascending or descending. If an incoming order does not provide the expected value then it should be suspended and the job chain waits for an order with the expected value to arrive. After arrival of an order with the expected value all suspended orders should then be checked to see if they provide the expected value required for the successor order. 
• Solution Outline:
  • A single job is added to at the top start of a job chain that will. This job:
    • check checks for exepected values and suspend suspends incoming orders until an order with the expected value arrives,
    • move orders moves orders that match expected values to the next job node in the job chain and recheck the rechecks the expected value of any suspended orders.
• References
  • SourceForge Forum Discussion Thread

...

• The solution implements a job expect that can be added to at the top start of any a job chain.
  • This job implements a spooler_process() function that checks if an order matches the expected value and that suspends non-matching orders.
    • Expected values can be provided by incoming orders:
      • using parameters:
        • each incoming order is assumed to provide a parameter that contains the expected value. 
          • Example for use with oder parameter:

            • business_date = 2015-11-01
        • the name of the incoming orders' parameter is specified as the value of the expect job's control_order_expected_parameter parameter.
          • Example for use with expect job parameter: 
            • control_order_expected_parameter= business_date
      • without parameters: 
        • the job will expect a numeric order ID to be provided and increments the order ID to calculate the next expected value.
    • For expected values a function has to be provided that calculates the next expected value:
      • the function is specified by the expect job's control_order_expected_value_function parameter. The function has been implemented with JavaScript code that returns the next expected value according to the current value that is provided with the currentValue variable:
        • Example for numeric calculation: 
          • parseInt(currentValue) + 1
          • Explanation: the function parses the current expected numeric value and returns the incremented value.
        • Example for date calculation:
          • (new Date( (new Date(currentValue)).setDate( (new Date(currentValue)).getDate()+1 ) )).toISOString().substring(0,10);
          • Explanation: the function accepts the current date value, adds one day and returns the date in ISO format, e.g. 2015-11-01
    • A default value can be provided for the expected values that is used to check the expected value of the first incoming order:
      • the default value can be specified by the expect job's control_order_expected_default_value parameter.
        • Example for use with numeric default value: 
          • control_order_expected_default_value = 1
        • Example for use with date default value: 
          • control_order_expected_default_value = 2015-11-01
      • without specification of a default value the value of the first order processed by the expect job will be used as a default.
  • This job is configured for a single task, i.e. it executes incoming orders sequentially.
  • Having received an order with the expected value this job moves that order to the next job node in the job chain and activates any suspended orders for rechecking to see if they provide the next expected value.
• The solution automatically creates a control order that is used to store the next expected value.
  • The name of the control order is generated from the "control_order_" prefix and the state that the expect job node is assigned.
  • The control order is suspended and not processed by subsequent job nodes. Its only purpose is to carry the next expected value for use of the solution with JobScheduler Agents and cluster members.
• The sample makes use of a job chain job_chain_expected_orders This chain includes the job nodes for the expect job and a hello job. The job chain accepts ad hoc orders that are added using JOC and can easily be modified to watch for incoming files and create an order for each file.
• Hint: to re-use the expect job you can:
  • store the job in a central folder and reference the job in individual job chains.
  • move the job's JavaScript code to a central location and use a corresponding an appropriate <include> element for individual job scripts.

...