Introduction
This article gives an overview of the JobScheduler Monitoring Interface. This solution provides an efficient means of of monitoring JobScheduler objects such as Jobs, Job Chains and Orders and forwarding notifications to system Monitors such as op5, Nagios or icing. This solution is available with JobScheduler General Availability Release 1.8 onwards.
The most important features of this solution are:
- JobScheduler: carries out a two step process around the interface:
- Detecting errors: A Job running at regular intervals - typically every 2 minutes - analyses the History Log information recorded by the JobScheduler in the database and notes a predefined set of information about the Job Scheduler objects being monitored. The information noted is typically whether tasks have been completed and whether errors or warnings have been logged. This job then writes this information in a separate Notifications database table.
- Sending alerts: A second Job is responsible for sending the alerts to the relevant System Monitor. This job is also run at regular intervals, analyzing the Notifications database tables. It then carries out a predefined action for each item it finds in the table. Typical actions would be informing a particular monitor that a particular type of event has occurred, such as the successful completion of an order or a job ending in error.
- JobScheduler: The solution architecture allows analysis of the Log History of more than one JobScheduler using the database specified. It may also be configured to monitor more that one database.
- System Monitors: the JobScheduler is able to connect to more than one System Monitor at the same time.
Monitoring Definitions
The following definitions apply for the monitoring systems:
Definition | Description |
---|---|
System Monitor | A System Monitor is an instrument to inform a Service Desk (e.g. 1st Level Support) about incidents in IT systems. It does not analyze incidents, but merely information about incidents, in order to be able to forward and scale this information. |
Passive Checks | Passive checks are sent remotely from an external host (from the point of view of the System Monitor) to the Monitor. Otherwise, checks that are carried out periodically by the System Monitor itself are called active checks. |
Alerting | An Alert is a message about an event. An Alert does not provide all the information about an event, but it informs about the existence of the event. An Alert can be either positive or negative. |
Notification | The notification of a specific Alert. Notification will not be provided for every Alert, just the ones that are so configured will be notified. Notifications are therefore a subset of the Alerts and can also be either positive or negative. |
Acknowledgment | Is the confirmation of an alert and it has the meaning that the alert has been seen and/or is known and that appropriate action is being taken. An acknowledgment is always manually executed. This means that there is always someone that has realized there is a Critical service and this person acknowledges the services (usually by the Service Desk or 1st Level Support). It is never an automatized step. |
Benefits
The benefits of the new solution are:
- No changes have to be made to your existing JobScheduler configuration (Jobs, Job Chains, etc.) in order to get this solution working. You add the Job Chains required for the monitoring but do not have to modify your current ones.
- The whole architecture lies on the JobScheduler side and the solution is therefore independent of the monitor that the Alerts are sent to. The solution works for every monitor that can receive passive checks.
- Processing of Jobs and Job Chains in JobScheduler is not affected or modified by the monitoring, neither from the point of view of performance nor that of stability.
- This solution makes very detailed information available for the System Monitors. JobScheduler logs very exactly and this information can be sent as a Passive Check to the relevant Monitoring Service if required.
- Errors of a critical nature are immediately recognized in the System Monitor. The JobScheduler has initially access to all the log information and can be configured to filter this information very exactly before forwarding it to the relevant System Monitor Service. This feature allows the Service Desk to be able to set priorities immediately when, for example, recovering errors. It is unlikely that a performance error would be given the same priority as an error in document processing. This feature is illustrated in the following diagram:
Functionality
Functionality | Description |
---|---|
Job Chain and Order Monitoring | This solution allows Job Chains in JobScheduler to be monitored by way of the Orders that trigger these Job Chains. |
History Notifications | Not only can critical alerts be monitored, but also positive ones. The history of a specific service can be monitored to see exactly if a specific work-flow has been executed and what result it gave. |
Performance measurement (Timer) | Timers can be used to measure the performance of Job Chains. These can be used to send a warning alert to a System Monitor if a Job Chain takes more that a predefined time to complete. |
Acknowledgment | Acknowledgments sent in response to critical alerts sent out by a System Monitor can be used to add Orders to the JobScheduler, so that the JobScheduler does not send more notifications about a service to the System Monitor. |
Monitoring example - op5 Monitor
The following example illustrates use of the JobScheduler Monitoring interface with the op5 Monitor. In the example, 3 checks (in op5 Monitor they are called services) have been defined for the JobScheduler monitoring. Different Job Chains in JobScheduler can send notifications to the same check, so that it is not necessary to create checks for each individual Job Chain, which could become extremely complex. Instead, results have been grouped in three categories:
- JobScheduler Monitoring Errors: Job Chains that end with an error are sent to this service. The last error notification is shown in the column "STATUS INFORMATION".
- JobScheduler Monitoring Success: Job Chains that end with success, that is with a positive notification, are sent to the monitoring system. To be exact, the history of a specific Job Chain is monitored to see whether a specific work-flow has been executed or not. The last success notification is shown in the column "STATUS INFORMATION".
- JobScheduler Monitoring Performance: Here timers are used to measure the performance of a Job Chain. If a Job Chain takes too long to end, a warning alert will be sent to the System Monitor. The information about the expired timer is shown in the column "STATUS INFORMATION".
Installation
See JobScheduler Monitoring Interface - Prerequisites and Installation
Configuration
JobScheduler - SystemMonitorNotification files
Location: <scheduler_install>/config/notification
File | Description |
---|---|
SystemMonitorNotification_v1.0.xsd | XML Schema file that defines which values are allowed in your XML files for the JobScheduler monitoring. That means, you just have to modify your |
SystemMonitorNotification_<MonitorSystem>.xml | Configuration file for each System Monitor.
|
| Configuration file for all System Monitors.
This file is optional and just has to contain the definitions of the |
SystemMonitorNotification Elements
The configuration element descriptions are organized into the following major categories:
Element | Element description | Description |
---|---|---|
SystemMonitorNotification | Top Level Element | Configuration for notifications to a system monitor |
Notification | Once or more inside a SystemMonitorNotification element | Specifies a system monitor notification that includes a command line invocation and the JobScheduler objects |
Timer | Optional, once or more inside a SystemMonitorNotification element | Performance measurement definition |
SystemMonitorNotification
SystemMonitorNotification
support the following attributes:
Note:
- attribute
system_id
in case of the
SystemMonitorNotificationTimers.xml
the value of this attribute is not important and can have any value.
e.g.:
timers
Attribute | Usage | Description |
---|---|---|
system_id | required | System Monitor identifier. |
SystemMonitorNotification / Notification
The following elements may be nested inside a Notification
element:
Element | Element description | Description |
---|---|---|
NotificationMonitor | Once inside a Notification element | Specifies the System Monitor interface that is being used for messages: either by a Plugin Interface or by command line invocation |
NotificationObjects | Once inside a Notification element | Specifies the JobChains and the Timers definitions |
SystemMonitorNotification / Notification / NotificationMonitor
NotificationMonitor
support the following attributes:
Note:
- attributes
service_name_on_error
andservice_name_on_success
- at least one of these attributes must be configured
- both attributes can be configured together
Attribute | Usage | Description |
---|---|---|
service_name_on_error | Optional | This setting specifies the service that is configured in the Service Monitor for messages of job runs with errors and for job recovery messages. The service name must match the corresponding setting in the System Monitor. |
service_name_on_success | Optional | This setting specifies the service that is configured in the Service Monitor for receiving informational messages on successful job runs. The service name must match the corresponding setting in the System Monitor |
service_status_on_error | Optional | This setting specifies the service status code for error messages. Default: |
service_status_on_success | Optional | This setting specifies the service status code for success messages Default: |
One of the following elements must be nested inside a NotificationMonitor
element:
Element | Element description | Description |
---|---|---|
NotificationInterface | Optional, once inside of NotificationMonitor element | Plugin Interface to be executed for System Monitor notification |
NotificationCommand | Optional, once inside of NotificationMonitor element | Command line to be executed for System Monitor notification |
SystemMonitorNotification / Notification / NotificationMonitor / NotificationInterface
NotificationInterface
support the following attributes:
Attribute | Usage | Description |
---|---|---|
monitor_host | Required | This setting specifies the hostname or ip address of System Monitor host. |
monitor_port | Required | This setting specifies the TCP port that the System Monitor would listen to. |
monitor_password | Optional | This setting specifies the password configured in the ncsa.cfg file used by NSCA. |
monitor_connection_timeout | Optional | This setting specifies the connection timeout in ms. Default: |
monitor_response_timeout | Optional | This setting specifies the NSCA response timeout in ms. |
monitor_encryption | Optional | This setting specifies that the communication with the System Monitor is encrypted. By default no encryption is used.
|
service_host | Required | This setting specifies the name of the host that executes the passive check. The name must match the corresponding setting in the System Monitor. |
plugin | Optional | Default: com.sos.scheduler.notification.plugins.notifier.SystemNotifierSendNscaPlugin |
SystemMonitorNotification / Notification / NotificationMonitor / NotificationCommand
NotificationCommand
support the following attributes:
Attribute | Usage | Description |
---|---|---|
plugin | Optional | Default: com.sos.scheduler.notification.plugins.notifier.SystemNotifierProcessBuilderPlugin |
SystemMonitorNotification / Notification / NotificationObjects
One of the following elements must be nested inside a NotificationObjects
element:
Element | Element description | Description |
---|---|---|
JobChain | Optional, once or more inside of NotificationObjects element | Restricts notifications for job chains |
Timer | Optional, once or more inside of NotificationObjects element | Restricts notifications for performance checks (Timer) |
SystemMonitorNotification / Notification / NotificationObjects / JobChain
JobChain
support the following attributes:
Attribute | Usage | Description |
---|---|---|
notifications | Optional Integer | Specifies the number of notifications that are sent to a System Monitor. Default: |
scheduler_id | Optional | Notifications are restricted to the JobScheduler instance with the given identification. By default notifications will be sent for all JobScheduler instances that would log into the same database. Regular expression can be used. |
name | Optional | Job chain name including possible folder names. Regular expression can be used. |
step_from | Optional | Restricts notifications for job chains to a sequence of job nodes that are specified with the step_from and step_to attributes. |
step_to | Optional | Restricts notifications for job chains to a sequence of job nodes that are specified with the step_from and step_to attributes. |
excluded_steps | Optional | Specifies the steps which will be excluded from the analysing (separated by semicolon) |
SystemMonitorNotification / Notification / NotificationObjects / Timer
Timer support the following attributes:
Attribute | Usage | Description |
---|---|---|
notifications | Optional Integer | Specifies the number of notifications that are sent to a System Monitor. Default: |
name | Optional | Correspondence to Timer name setting defined in the SystemMonitorNotification / Timer element |
notify_on_error | Optional Boolean | Send timer check notification when the configured job chain contains the error notifications. Default: |
SystemMonitorNotification / Timer
The following elements must be nested inside a Timer
element:
Element | Element description | Description |
---|---|---|
JobChain | Once or more inside of Timer element | Restricts notifications for job chains |
Minimum | Optional or once inside of Timer element | Minimum required time consumption for job or job chain execution. Allows script code to be executed that returns the minimum execution time required in seconds. |
Maximum | Optional or once inside of Timer element | Maximum allowed time consumption for job or job chain execution. Allows script code to be executed that returns the maximum execution time required in seconds. |
Timer
support the following attributes:
Attribute | Usage | Description |
---|---|---|
name | Required | Correspondence to Timer used in the The name must be unique across all timers definitions. |
SystemMonitorNotification / Timer / JobChain
JobChain
support the following attributes:
Attribute | Usage | Description |
---|---|---|
scheduler_id | Optional | Notifications are restricted to the JobScheduler instance with the given identification. By default notifications will be sent for all JobScheduler instances that would log into the same database. Regular expression can be used. |
name | Optional | Job chain name including possible folder names. Regular expression can be used. |
step_from | Optional | Restricts checks for job chains to a sequence of job nodes that are specified with the step_from and step_to attributes. |
step_to | Optional | Restricts checks for job chains to a sequence of job nodes that are specified with the step_from and step_to attributes. |
SystemMonitorNotification / Timer / Minimum
The following elements must be nested inside a Minimum
element:
Element | Element description | Description |
---|---|---|
Script | Once inside of Minimum element | Script code in one of the supported languages |
SystemMonitorNotification / Timer / Maximum
The following elements must be nested inside a Maximum
element:
Element | Element description | Description |
---|---|---|
Script | Once inside of Maximum element | Script code in one of the supported languages |
SystemMonitorNotification / Timer / Minimum|Maximum / Script
Script
support the following attributes:
Attribute | Usage | Description |
---|---|---|
language | Required | Script language name Supported languages:
|
The Script element can contains
- fixed value
- calculation based on the job/order parametes
Fixed value
Fixed value is the duration time in seconds for the specific Minimum
or Maximum
definition
Calculation
The calculation result is the time in seconds for the specific Minimum
or Maximum
definition.
This example calculate the execution time dependend of the %file_size%
parameter, that was set by a specific job (see the example below)´.
This example job calculate and create a new order parameter file_size
.
To store the parameters into database (table SCHEDULER_MON_RESULTS
) :
- set the
scheduler_notification_result_parameters
parameter (see job documentationjobs/JobSchedulerNotificationStoreResultsJob.xml
) - set the
com.sos.scheduler.notification.jobs.result.StoreResultsJobJSAdapterClass
as monitor
Message
Usage
The Message can be configured on the following parent nodes as CDATA element :
SystemMonitorNotification / Notification / NotificationCommand
SystemMonitorNotification / Notification / NotificationInterface
The Message can contains:
- fixed values
- variables
Example: <![CDATA[ scheduler id = %MON_N_SCHEDULER_ID% ]]>
Variables
All variables must be defined by using of the %<variable name>%
syntax.
The order of the substitution the variables values is:
- Table variables.
- Service variables.
- OS environment variables.
Table variables
Service variables
OS environment variables
All existing system variables can be defined by message with the syntax like %<variable name>%
(Windows/Unix)
.
Examples
Notification environment variables
The default com.sos.scheduler.notification.plugins.notifier.SystemNotifierProcessBuilderPlugin
plugin used by the SystemMonitorNotification / Notification / NotificationCommand
element sets the following variables as environment variables:
Table variables
Service variables
These variables can be used when the NotificationCommand calls the notification client not directly, but a shell script, that make the logical implementation for sending of the notification messages.
Table variables
Service variables
Examples
Examples OP5
NotificationInterface
Here is an except of an XML file used for notifying a specific System Monitor (OP5 Monitor) and using NotificationInterface:
NotificationCommand
Here is an except of an XML file used for notifying a specific System Monitor (OP5 Monitor) and using NotificationCommand on Windows:
Examples Zabbix
NotificationCommand
Here is an except of an XML file used for notifying a specific System Monitor (Zabbix Monitor) and using NotificationCommand
JobScheduler - Job Chains
WORK IN PROGRESS
Use Cases
Recoverable Errors
Initial Situation: A Job Chain is triggered by directory monitoring. That is, when a certain file comes in a monitored folder, the Job Chain starts.
Problem: The Job Chain ended with error.
Handling: The System Monitor will be notified to the service related to the Job Chain with the message error. If a new execution of the Job Chain from a new file end without errors, does not mean that the error is recovered, since the file that has been processed is now another one. That is, the error message at the System Monitor will stay till the same file is again placed in the monitored directory and the Job Chain ends without errors.
Configuration:
- XML
CheckConfigurationHistory.xml
: Indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor. - XML
SystemMonitorNotification.xml
: Specify the name of the Service (in the System Monitor) and specify that it is about aservice_name_on_error
since you want to have the control when the Job Chain ends in an error. - System Monitor: Services in the System Monitor have to be configured and named the same way as in the XML file above
SystemMonitorNotification.xml
.
Workflow Execution takes too long
Initial Situation: A Job Chain is triggered and it could not end, it hanged in a step, taking then longer than expected.
Problem: Execution time was too long
Handling: A timer for this Job Chain is set and the System Monitor will be notified about it. The expiration times for the Job Chains are configured with enough time for processing, that means, this is usually used for cases where the Job Chain hanged in a specific step.
Configuration:
- XML
CheckConfigurationHistory.xml
: As in the example above, indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor. Moreover, specify the timer for this specific job chain and the function to calculate the expiration time for the timer. - XML
SystemMonitorNotification.xml
: As in the example above, specify the name of the Service (in the System Monitor) and specify that it is about aservice_name_on_error
since you want to have the control when the Job Chain ends in an error. Moreover and essential for this particular case, specify how many times the timer should notify your System Monitor about the expiration of a timer. - System Monitor: As in the example above, Services in the System Monitor have to be configured and named the same way as in the XML file above
SystemMonitorNotification.xml
.
SFTP connection refused
Initial Situation: There is a Job Chain that uses SFTP for transferring files. You have a setback configured in this step of the Job Chain, so that if the connection to the SFTP server fails, this step is retried after some time.
Problem: The SFTP server is not available anymore.
Handling: The System Monitor will be notified to the service related to the Job Chain with the message error. However, you don't want to have a bunch of notifications for a Job Chain when is an external factor, the connection to the SFTP Server, what is producing the error.
Configuration:
- XML
CheckConfigurationHistory.xml
: As in the example above, indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor. - XML
SystemMonitorNotification.xml
: As in the example above, specify the name of the Service (in the System Monitor) and specify that it is about aservice_name_on_error
since you want to have the control when the Job Chain ends in an error. Moreover and very important in this case, specify how many times this Job Chain should notify your System Monitor about the error connecting to the SFTP Server. You can usestep_from
andstep_to
for that in order to reduce the number of notifications for this specific step. - System Monitor: As in the example above, Services in the System Monitor have to be configured and named the same way as in the XML file above
SystemMonitorNotification.xml
.
Thresholds
Initial Situation: For example, a specific number of Workflow Executions have to be executed successfully till some specific time. That is, a specific value has to be monitored in order to determine if this quote was reached.
Handling: A new service for History is configured, so that the workflow executions (Job Chains in the JobScheduler vocabulary) send the information that they were executed and finished to the System Monitor.
Configuration:
- XML
CheckConfigurationHistory.xml
: As in the example above, indicate the ID of the JobScheduler and the name of the Job Chain you want to monitor. - XML
SystemMonitorNotification.xml
: Specify the name of the Service (in the System Monitor) but now specify that it is about aservice_name_on_success
since you want to have the control when the Job Chain ends in an success, and not only when it ends on error. - System Monitor: As in the example above, Services in the System Monitor have to be configured and named the same way as in the XML file above
SystemMonitorNotification.xml
.
Acknowledgement
Initial Situation: An alert for a Service has been sent to the System Monitor and a Mail has been sent to the Service Desk (Support Team) notifying about it.
Handling: The problem is well known by the Service Desk and the "acknowledge" the problem. Through the acknowledgement JobScheduler will be notified to and will not send any more notification for this Service to the System Monitor till the Service is again recovered.
Configuration:
- System Monitor: The step of notifying JobScheduler through an acknowledgement in the System Monitor is an execution of a script. This is nothing else than a notification, like sending a mail for instance, but instead, another action is executed, which is the execution of the script that contacts JobScheduler and add an order to the JobChain
ResetNotifications
described above.