Introduction
Dependencies between workflows include that JS7 - Notice Boards are used by a JS7 - PostNotices Instruction to create Notices that other workflows are waiting for using the JS7 - ExpectNotices Instruction and JS7 - ConsumeNotices Instruction.
- By default such dependencies remain in place for an arbitrary duration until they can be resolved.
- Example:
- Workflow A includes a PostNotices Instruction to create a Notice.
- Workflow B includes a ConsumeNotices Instruction to expect and to remove the Notice.
- Assume that Workflow A is delayed - possibly for days - and that no related Notice is created. This makes the order in Workflow B wait until the Notice arrives. Should the order in Workflow A be cancelled then the order in Workflow B will wait endlessly unless it is cancelled too. The reason being that Notices are based on Daily Plan dates and act on orders that share the same Daily Plan date as the Notice.
For some users this is the desired behavior, for others it is not: users handle dependencies in the JS7 - Daily Plan individually.
- Some users want dependencies to remain in place until they can be resolved.
- Example:
- Assume Workflow A to load reporting data to a Data Warehouse on a daily basis. Assume Workflow B to run reporting jobs for the Data Warehouse.
- If reporting data are sequential and build upon each other then the order in Workflow B has to wait until reporting data provided by Workflow A are present before creating a report.
- In this scenario users want Workflow B to be delayed for hours or days until data from Workflow A are ready as the sequential nature of data delivery forces this behavior.
- Example:
- Some users want dependencies to be dropped after a certain period.
- Example:
- Assume a scenario of classic batch processing when Workflow A loads data to a database that are processed by Workflow B with both workflows running on a daily basis.
- If Workflow A is late within a day then the order for Workflow B should wait. However, if Workflow A exceeds the 24 hours' period of the Daily Plan it should be cancelled as the same Workflow A is executed by a new order on the next day. In this scenario Workflow B should not wait longer for Workflow A than the Daily Plan's period.
- Example:
The following explanations focus on the scenario to clean up the Daily Plan from past dependencies of a previous day using Unix shell scripts.
Cleanup
Cleanup includes two operations for which shell scripts are available:
- Drop Orders scheduled for past Daily Plan dates.
cancel-orders.sh
: see JS7 - How to cancel orders using the REST Web Service API from the Shell
- Drop Notices created for past Daily Plan dates.
delete-notices.sh
: see JS7 - How to delete notices using the REST Web Service API from the Shell
Both operations to clean up the Daily Plan can be combined in a workflow for automated execution on a regular basis.
Workflow
The workflow implements cleanup from two shell jobs introduced with the indicated articles.
- Job
Cancel-Orders
using thecancel-orders.sh
shell script - Job
Delete-Notices
using thedelete-notices.sh
shell script
Download the sample workflow (upload .json): pdwCleanupDailyPlan.workflow.json
Explanation:
- The workflow makes use of the
Default
job resource, see JS7 - Job Environment Variables and JS7 - Job Resources.- It is not required to use the job resource that is available from JS7 - Download.
- However, the job resource simplifies parameterization of the job.
- Error handling includes to use a JS7 - Try-Catch Instruction that will continue processing with the Catch Block should one of the jobs in the Try Block fail.
Job: Cancel-Orders
The job makes use of the cancel-order.sh
shell script to cancel orders waiting for past notices from JS7 - Notice Boards.
For details see JS7 - How to cancel orders using the REST Web Service API from the Shell.
Explanations:
- Parameterization of the
cancel-orders.sh
script includes:-url="${JS7_JOC_URL}
: The JOC Cockpit URL is provided from an environment variable that users can specify with theDefault
job resource.-user=root
,--password=root
: The example makes use of user/password authentication. A number of ways are offered for- JS7 - Use of Credential Store with Shell Jobs: explains how to read accounts and passwords from a credential store.
- JS7 - Identity Services: offer a number of authentication methods such as user/password, LDAP, certificates etc.
- JS7 - How to connect to JOC Cockpit using the PowerShell Module: provides examples for use of different authentication methods that similarly apply to shell jobs.
--controller-id="${JS7_CONTROLLER_ID}"
: The Controller ID is provided from an environment variable available with theDefault
job resource.--date-to="$(TZ="${JS7_AGENT_TZ}" date --date="1 day ago" +'%Y-%m-%d')T00:00:00"
: Specifies the date and time to which orders have been scheduled that will be cancelled.- The time zone is provided from the
JS7_AGENT_TZ
environment variable available with theDefault
job resource. - The syntax
TZ=<time-zone> date ...
executes thedate
command for the time zone specified. For time zone identifiers see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
- The time zone is provided from the
--time-zone="${JS7_AGENT_TZ}"
: Specifies that the Agent's time zone will be used. The value of this option has to match the time zone assumed for the--date-to
option.--states=SCHEDULED,WAITING
: Specifies that orders in the SCHEDULED and WAITING state should be cancelled.
Job: Delete Notices
The job makes use of the delete-notices.sh
shell script to delete notices from JS7 - Notice Boards.
For details see JS7 - How to delete notices using the REST Web Service API from the Shell
Explanations:
- Parameterization of the
delete-notices.sh
script includes:- For parameters related to access and authentication with JOC Cockpit see chapter Job: Cancel-Orders
--date="$(TZ="${JS7_AGENT_TZ}" date --date="1 day ago" +'%Y-%m-%d')"
: Explanations for job Cancel-Orders apply. The difference being that --date expects a date without time as it specifies the day for which notices have been created that should be deleted.--time-zone="${JS7_AGENT_TZ}"
: Specifies that the Agent's time zone will be used. The value of this option has to match the time zone assumed for the--date
option.
Further Resources
- JS7 - How to cancel orders using the REST Web Service API from the Shell
- JS7 - How to delete notices using the REST Web Service API from the Shell