How to trigger a job or a job chain based on events?

Events

Question: How can I trigger a job or a job chain based on a (series of) event(s)?

Introduction

In some cases it is necessary to trigger a job (or a job chain) based on (multiple) events (other jobs, monitored files...) which might even occur on different hosts.

JobScheduler provides a solution to set off events and to handle these events on an active instance of the event service JobScheduler.
You find the complete documentation in scheduler_events.pdf - Documentation. Especially how to define event handlers with JOE is described in chapter 4.1.3 XML Event Handlers
To show the usage of events we will draw up a simple scenario:

1. start a single job which will do something,
2. watch a folder for an incoming file with the extension ".txt" and start at job to process and/or move this file
3. watch the same folder for an incoming file with the extension ".pdf" and start a job chain to process and move this file

After these three actions are executed, in a randomly sequence, and successfully finished, another job chain must be started and all events must be cleared.

For this example, two different JobScheduler instances may be configured.
An event service (or event supervisor) and a Workload JobScheduler which is registered with that event service.
The event service is able to receive events from other JobScheduler instances on different servers and platforms.
It is possible also to have the event service and the Workload JobScheduler running in the same JobScheduler instance.
If different JobScheduler instances were used for eventing then it is required that these instances can communicate.

Requirements

The JobScheduler must run using a database. The events are stored permanently in the table "SCHEDULER_EVENTS". To customize an event handler some experience with XML and XSLT are required.

Instructions

• Unzip all files of events.zip or events_windows.zip into the ./config/live folder of your Workload JobScheduler installation. After unzipping you will have in the subfolder samples an subfolder events with the required JobScheduler objects to run the example.
• Edit ./config/scheduler.xml of the Workload JobScheduler and configure a JobScheduler as a Supervisor. The Supervisor is an attribute of the tag config. See Supervisor.
• In the installation directory of your workload JobScheduler create a folder files with two subfolders in and processed
• Unzip all files of events_supervisor.zip into the ./config folder of your instance of the event service JobScheduler installation. After unzipping you will have in the subfolder ./config/live/sos a subfolder events with all the objects required to run the event service. And a folder ./config/events that is used by the event handler.
• Edit config/events/example.event_class.xsl on the active instance of the event service JobScheduler and adjust the host and port parameters to those of the Workload JobScheduler. Copy the file to the folder ./config/events.
   

          <xsl:template match="events[event[@job_name='simple_shell_job'] and event[@job_chain='txt_chain'] and event[@job_chain='pdf_chain']]">
            <xsl:call-template '''name="run_job"'''>
              <xsl:with-param name="job">'''samples/events/done_job<'''/xsl:with-param>
              <xsl:with-param '''name="host">localhost'''</xsl:with-param>
              <xsl:with-param '''name="port">4444'''</xsl:with-param>
            </xsl:call-template>
            <remove_event>
              <event event_class="example"/>
            </remove_event>
          </xsl:template>

The config/events/example.event_class.xsl file, which is shown above, customizes an event handler which will start the job samples/events/done_job if all three expected events have arrived.

In the next steps these events will be initiated:

• Open the JobScheduelr Operations GUI of the Workload JobScheduler in your browser using http://scheduler_host:scheduler_port
• Find the job samples/events/simple_shell_job and click on it:
   

          <job>
            <script language="shell">
                <![CDATA[
          # run any shell script or command(s)
          echo hello world!
          
          # submit event to active instance of the eventservice JobScheduler
          $\{SCHEDULER_HOME\}/bin/jobscheduler_event.sh -x $? -e "example"
                ]]>
            </script>
            <run_time/>
          </job>

• In the Job menu press "Start task now" (1st event)
• Put a .txt file in the files/in directory. This should cause samples/events/file_job to run and move the file to the files/processed directory. (2nd event):
   

          <job order="yes">
              <script language="shell">
                  <![CDATA[
          # run any shell script or command(s)
          echo processing file $SCHEDULER_PARAM_SCHEDULER_FILE_PATH
          
          mv $SCHEDULER_PARAM_SCHEDULER_FILE_PATH files/processed
          
          # submit event to Supervisor Job JobScheduler
          $\{SCHEDULER_HOME\}/bin/jobscheduler_event.sh -x $? -e "example"
                  ]]>
              </script>
              <run_time/>
          </job>

• Put a .pdf file in the files/in directory.
  This will cause samples/events/file_job to run and move the file to the files/processed directory. (3rd event)

When all three events (coming from simple_shell_job, .txt file, .pdf file) were thrown, the job samples/events/done_job will immediately start and execute the requested action: delete the files from the files/processed directory.

        <job name="done_job">
            <script language="shell">
                <![CDATA[
        echo content of files/processed:
        ls -la files/processed
        
        echo deleting contents of files/processed
        rm -f files/processed/*
                ]]>
            </script>
            <run_time/>
        </job>

How it works

The job samples/events/simple_shell_job calls the script

  {{$\{SCHEDULER_HOME\}/bin/jobscheduler_event.sh -x $? -e "example"}}

to signal an event of call "example" to the active instance of the even tservice JobScheduler.
The -x switch sets the exit code of the previous shell command (echo in this case), and the -e switch sets the event class as properties of the event.
Other properties (which do not need to be set as parameters for the script jobscheduler_event.sh) include the name of the job that created the event, the name of the job chain (if the job was called in a chain), name of the order etc.

Similarly the samples/events/file_job creates events that are collected in the active instance of the event service JobScheduler.
The samples/events/file_job is used in two separate job chains: samples/events/txt_chain (which triggers .txt files) and samples/events/pdf_chain (which triggers .pdf files).
Thus, two more events are submitted to the JobScheduler with the event class example and with two different job chain properties.

Each time the active instance of the event service JobScheduler receives an event it runs the JobScheduler_event_service job .
The job will then process all event handlers that follow the naming scheme:

 *[_job_name_].*.job.xsl will be run for events created by the job [_job_name_]
 *[_job_chain_name_].*.job_chain.xsl will be run for events created by a job in the job chain [_job_chain_name_]
 *[_event_class_name_].*.event_class.xsl will be run for events with the event class [_event_class_name_]

In this example, all events are assigned to the event class example, so for each event the event handler (i.e.the XSLT stylesheet) config/events/example.event_class.xsl is executed and parsed by a running job of the active instance of the event service JobScheduler.

This stylesheet processes an xml object that contains all events currently known to the active instance of the event service JobScheduler, e.g.:

   <events current_date="2011-08-06 11:14:01" expiration_date="2011-08-06 23:14:01">
     <event created="2011-08-06 11:13:29" event_class="example"
       event_id="0" exit_code="0" expires="2011-08-06 23:13:30"
       job_chain="" job_name="simple_shell_job" order_id=""
       remote_scheduler_host="wilma" remote_scheduler_port="4444" JobScheduler_id="scheduler.supervisor"/>
     <event created="2011-08-06 11:14:01" event_class="example"
       event_id="0" exit_code="0" expires="2011-08-06 23:14:01"
       job_chain="txt_chain" job_name="file_job"
       order_id="files/in/sample2.txt" remote_scheduler_host="wilma"
       remote_scheduler_port="4444" JobScheduler_id="scheduler.supervisor"/>
   </events>

The event handler (stylesheet) has a template that tries to find 3 events, one from the simple_shell_job job, and one from each of the file handling job chains:

   <xsl:template match="events[
                     event[@job_name='simple_shell_job'] and
                     event[@job_chain='txt_chain'] and event[@job_chain='pdf_chain']
                     ]">

If these events are not found, then nothing happens. The events remain in the active instance of the event service JobScheduler.
If the events are found then the event handler stylesheet returns two commands to the active instance of the event service JobScheduler:

The first command starts the samples/events/done_job on the Workload JobScheduler:

     <xsl:call-template name="run_job">
       <xsl:with-param name="job">samples/events/done_job</xsl:with-param>
       <xsl:with-param name="host">localhost</xsl:with-param>
       <xsl:with-param name="port">4444</xsl:with-param>
     </xsl:call-template>

The run_job template is a template from the imported stylesheet scheduler_event_functions.xsl.inc which is provided to facilitate creating (basic) JobScheduler commands.

The second command cleans up the current set of events of the event class example to avoid triggering samples/events/done_job again before all 3 events have occurred:

     <remove_event>
       <event event_class="example"/>
     </remove_event>

The scheduler_event_functions.xsl.inc stylesheet

It is recommended to include scheduler_event_functions.xsl.inc into your own event handler stylesheets by adding:

     <xsl:import href="scheduler_events.xsl.inc"/>

to the stylesheet. (see example.event_class.xsl)

This included stylesheet will ensure:

• that a root level <commands> tag (element) is created in the resulting xml and
• that the stylesheet will return a valid (and well formed) xml result (returning nothing would result in an error)

Examples for match statements in event handlers

        {{<xsl:template match="events[event[@job_name='simple_shell_job' and @exit_code=0]]">}}

Matches if an event exists for job simple_shell_job with exit code 0.

        {{<xsl:template match="events[event[@job_name='simple_shell_job' and @exit_code>0]]">}}

Matches if an event exists for job simple_shell_job with an exit code greater than 0.

        {{<xsl:template match="events[event[@job_name='simple_shell_job'] and not(event[@job_name='file_job'])]">}}

Matches if an event exists for job simple_shell_job but no event for job file_job.

        {{<xsl:template match="events[event[@job_name='simple_shell_job'] or count(event[@event_class='other_events'])>=3]">}}
        

Matches if an event exists for job simple_shell_job or at least three events of event class other_events.

    {{<xsl:template match="events[event[@job_name='simple_shell_job']/params/param[@name='foo' and @value='bar'] ]">}}
  

Matches if an event exists for job simple_shell_job with an event parameter "foo" with the value "bar".

Downloads:

• scheduler_events.pdf - Documentation

• events.zip - job configuration files
• events_windows.zip - job configuration files (windows version)
• events_supervisor.zip - event handler files