Introduction

Dependencies between workflows include that JS7 - Notice Boards are used by a JS7 - PostNotices Instruction to create Notices that other workflows are waiting for using the JS7 - ExpectNotices Instruction and JS7 - ConsumeNotices Instruction.

• By default such dependencies remain in place for an arbitrary duration until they can be resolved.
• Example:
  • Workflow A includes a PostNotices Instruction to create a Notice.
  • Workflow B includes a ConsumeNotices Instruction to expect and to remove the Notice.
  • Assume that Workflow A is delayed - possibly for days - and that no related Notice is created. This makes the order in Workflow B wait until the Notice arrives. Should the order in Workflow A be cancelled then the order in Workflow B will wait endlessly unless it is cancelled too. The reason being that Notices are based on Daily Plan dates and act on orders that share the same Daily Plan date as the Notice.

For some users this is the desired behavior, for others it is not: users handle dependencies in the JS7 - Daily Plan individually.

• Some users want dependencies to remain in place until they can be resolved.
  • Example:
    • Assume Workflow A to load reporting data to a Data Warehouse on a daily basis. Assume Workflow B to run reporting jobs for the Data Warehouse.
    • If reporting data are sequential and build upon each other then the order in Workflow B has to wait until reporting data provided by Workflow A are present before creating a report.
    • In this scenario users want Workflow B to be delayed for hours or days until data from Workflow A are ready as the sequential nature of data delivery forces this behavior.
• Some users want dependencies to be dropped after a certain period.
  • Example:
    • Assume a scenario of classic batch processing when Workflow A loads data to a database that are processed by Workflow B with both workflows running on a daily basis.
    • If Workflow A is late within a day then the order for Workflow B should wait. However, if Workflow A exceeds the 24 hours' period of the Daily Plan it should be cancelled as the same Workflow A is executed by a new order on the next day. In this scenario Workflow B should not wait longer for Workflow A than the Daily Plan's period.

The following explanations focus on the scenario to clean up the Daily Plan from past dependencies of a previous day using Unix shell scripts.

Cleanup

Cleanup includes two operations for which shell scripts are available:

• Drop Orders scheduled for past Daily Plan dates.
  • cancel-orders.sh: see JS7 - How to cancel orders using the REST Web Service API from the Shell
• Drop Notices created for past Daily Plan dates.
  • delete-notices.sh: see JS7 - How to delete notices using the REST Web Service API from the Shell

Both operations to clean up the Daily Plan can be combined in a workflow for automated execution on a regular basis.

Workflow

The workflow implements cleanup from two shell jobs introduced with the indicated articles.

• Job Cancel-Orders using the cancel-orders.sh shell script
• Job Delete-Notices using the delete-notices.sh shell script

Download the sample workflow (upload .json): pdwCleanupDailyPlan.workflow.json

Explanation:

• The workflow makes use of the Default job resource, see JS7 - Job Environment Variables and JS7 - Job Resources.
  • It is not required to use the job resource that is available from JS7 - Download.
  • However, the job resource simplifies parameterization of the job.
• Error handling includes to use a JS7 - Try-Catch Instruction that will continue processing with the Catch Block should one of the jobs in the Try Block fail.

Job: Cancel-Orders

The job makes use of the cancel-order.sh shell script to cancel orders waiting for past notices from JS7 - Notice Boards.

For details see JS7 - How to cancel orders using the REST Web Service API from the Shell.

Explanations:

• Parameterization of the cancel-orders.sh shell script includes:
  • -url="${JS7_JOC_URL}: The JOC Cockpit URL is provided from an environment variable that users can specify with the Default job resource.
  • -user=root, --password=root: The example makes use of user/password authentication. A number of ways are offered for authentication and management of credentials:
    • JS7 - Use of Credential Store with Shell Jobs: Explains how to read accounts and passwords from a credential store.
    • JS7 - Identity Services: Offer a number of authentication methods such as user/password, LDAP, certificates etc.
    • JS7 - How to connect to JOC Cockpit using the PowerShell Module: Provides examples for use of different authentication methods that similarly apply to shell jobs.
  • --controller-id="${JS7_CONTROLLER_ID}": The Controller ID is provided from an environment variable available with the Default job resource.
  • --date-to="$(TZ="${JS7_AGENT_TZ}" date +'%Y-%m-%d')T00:00:00": Specifies the date and time to which orders have been scheduled that will be cancelled.
    • The time zone is provided from the JS7_AGENT_TZ environment variable available from the Default job resource.
    • The syntax TZ=<time-zone> date ... executes the date command for the time zone specified. For time zone identifiers see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
    • If the job is executed on 23rd of October 2023 then the parameterization evaluates to --date-to=2023-10-23T00:00:00 to indicate that any orders scheduled prior to this date should be cancelled.
  • --time-zone="${JS7_AGENT_TZ}": Specifies that the Agent's time zone will be used. The value of this option has to match the time zone assumed for the --date-to option.
  • --states=SCHEDULED,WAITING: Specifies that orders in the SCHEDULED and WAITING state should be cancelled, see JS7 - Order State Transitions.
• Error handling allows exit codes 0 and 3 to signal successful execution. Exit 3 signals that no orders have been found that match the criteria which is not considered an error.

Job: Delete Notices

The job makes use of the delete-notices.sh shell script to delete Notices from JS7 - Notice Boards.

For details see JS7 - How to delete notices using the REST Web Service API from the Shell

Explanations:

• Parameterization of the delete-notices.sh shell script includes:
  • For parameters related to access and authentication with JOC Cockpit see chapter Job: Cancel-Orders
  • --date="$(TZ="${JS7_AGENT_TZ}" date --date="1 day ago" +'%Y-%m-%d')": Explanations for job Cancel-Orders apply. The difference being that the --date expects a date without time as it specifies the day for which Notices have been created that should be deleted. In addition, the parameterization specifies to use previous day. If the job is executed on 23rd of October 2023 then the parameterization evaluates to --date=2023-10-22.
  • --time-zone="${JS7_AGENT_TZ}": Specifies that the Agent's time zone will be used. The value of this option has to match the time zone assumed for the --date option.
• Error handling allows exit codes 0 and 3 to signal successful execution. Exit code 3 signals that no notices have been found that match the criteria which is not considered an error.

Logging

Log output of the workflow can look like this:

Explanations:

• For both jobs the parameterization is logged.
• The jobs did not find orders/notices for cancellation/deletion. This fact is logged to the stderr channel but is not considered an error.

Further Resources

• JS7 - How to cancel orders using the REST Web Service API from the Shell
• JS7 - How to delete notices using the REST Web Service API from the Shell