...

• By default such dependencies remain in place for an arbitrary duration until they can be resolved.
• Example:
  • Workflow A includes a PostNotices Instruction to create a Notice.
  • Workflow B includes a ConsumeNotices Instruction to expect and to remove the Notice.
  • Assume that Workflow A is delayed - possibly for days - and that no related Notice is created. This makes the order in Workflow B wait until the Notice arrives. Should the order in Workflow A be cancelled then the order in Workflow B will wait endlessly unless it is cancelled too. The reason being that Notices are based on Daily Plan dates and act on orders that share the same Daily Plan date as the Notice.

For some users this is the desired behavior, for some others it is not: users handle dependencies in the JS7 - Daily Plan individually.

• Some users want dependencies to remain in place until they can be resolved.
  • Example:
    • Assume Workflow A to load reporting data to a Data Warehouse on a daily basis. Assume Workflow B to run a report from reporting jobs for the Data Warehouse.
    • If reporting data are sequential and build upon each other then the order in Workflow B has to wait until reporting data provided by Workflow A are present before creating a report.
    • In this scenario users want Workflow B to be delayed for hours or days until data from Workflow A are ready as the sequential nature of data deliveries delivery forces this behavior.
• Some users want dependencies to be dropped after a certain period.
  • Example:
    • Assume a scenario of classic batch processing when Workflow A loads data to a database that are processed by Workflow B with both workflows running on a daily basis.
    • If Workflow A is late within a day then the order for Workflow B should wait. However, if Workflow A exceeds the 24 hours' period of the Daily Plan it should be cancelled as the same Workflow A is executed by a new order on the next day. In this scenario Workflow B should not wait longer for Workflow A than the Daily Plan's period.

...

Cleanup includes two operations for which script solutions shell scripts are available:

• Drop Orders scheduled for past Daily Plan dates.
  • cancel-orders.sh: see JS7 - How to cancel orders using the REST Web Service API from the Shell
• Drop Notices created for past Daily Plan dates.
  • delete-notices.sh: see JS7 - How to delete notices using the REST Web Service API from the Shell

Both operations to clean up the Daily Plan can be combined in a workflow for automated execution on a daily regular basis.

Workflow

The workflow implements cleanup from two shell jobs introduced with the above indicated articles.

• Job Cancel-Orders using the cancel-orders.sh shell script
• Job Delete-Notices using the delete-notices.sh shell script

Download the sample workflow (upload .json): pdwCleanupDailyPlan.workflow.json

...

• Parameterization of the cancel-orders.sh script includes:
  • -url="${JS7_JOC_URL}: The JOC Cockpit URL is provided from an environment variable that users can specify with the Default job resource.
  • -user=root, --password=root: The example makes use of user/password authentication. A number of ways are offeredoffered  for
    • JS7 - Use of Credential Store with Shell Jobs: explains how to read accounts and passwords from a credential store.
    • JS7 - Identity Services: offer a number of authentication methods such as user/password, LDAP, certificates etc.
    • JS7 - How to connect to JOC Cockpit using the PowerShell Module: provides examples for use of different authentication methods that similarly apply to shell jobs.
  • --controller-id="${JS7_CONTROLLER_ID}": The Controller ID is provided from an environment variable available with the Default job resource.
  • --date-to="$(TZ="${JS7_AGENT_TZ}" date --date="1 day ago" +'%Y-%m-%d')T00:00:00": Specifies the date and time to which orders have been scheduled that will be cancelled.
    • The time zone is provided from the JS7_AGENT_TZ environment variable available with the Default job resource.
    • The syntax TZ=<time-zone> date ... executes the date command for the time zone specified. For time zone identifiers see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
  • --time-zone="${JS7_AGENT_TZ}": Specifies that the Agent's time zone will be used. The value of this option has to match the time zone assumed for the --date-to option.
  • --states=SCHEDULED,WAITING: Specifies that orders in the SCHEDULED and WAITING state should be cancelled.

...