Page History

...

• Resilience includes support for a number of outage scenarios with automated and manual recovery.
  

  Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JS-1518

   
• Outage Scenarios
  • Network Connection Loss
    • A connection loss between Master and Agent. The Master will retry attempts to establish the connection and to re-send requests for a number of times.
    • Connection loss includes that from the beginning the JobScheduler Master and Agent have no knowledge if the network connection failed or if a Master Service Failure occurred.
    • This scenario is intended for a connection failure that can be recovered by retry attempts to establish a connection, it is not intended to recover from an on-going network outage. 
  • Master Service Failure
    • Either a loss of the connection between Master and Agent that cannot be recovered within the number of retry attempts specified for the Network Connection Loss scenario
      • due to a server crash or
      • due to a JobScheduler Master crash.
    • Or an unplanned JobScheduler Master restart or server restart.
  • Database Connection Loss
    • A connection loss beetween Master and database:
      • for a JobScheduler Active Cluster this scenario includes a period of less than 120s during which a cluster member retries attempts to establish the connection.
      • for a JobScheduler Passive Cluster this scenario includes no restriction of duration, it can be configured to retry attempts to connect to the database endlessly.
        • factory.ini max_db_errors=0
    • Connection loss includes that the JobScheduler Master has no knowledge if the database service failed or if the connection failed.

...

• Reconciliation Scenario
  • Applies after a Network Connection Loss between Master and Agent.
  • If the connection can be re-established then running tasks are continued with the Agent, otherwise running tasks are killed.
• Objectives
  • If a Master were not available for a longer period then the Agent could not report back the execution history and log information for tasks. This would result in the fact that no information is available with the Master if the job execution has been successful or not.
  • The primary goal is to prevent duplicate simultaneous execution of jobs. Without further information from a Master the respective Agent instance cannot know if later on it will be contacted for re-execution of the same job (which would allow to continue a currently running task on an Agent) or if the Master will choose a different Agent (see Redundancy, Agent BundleCluster).
  • The secondary goal is to support re-establishing the communication between Master and Agent and to continue running tasks. Tasks that make use of the JobScheduler API cannot run independently from the Master and are delayed within the scope of this feature.

• Master/Agent Heartbeats

  • The Master and Agent send heartbeats to each other.
    • The Agent receives HTTP POST requests from the Master and will respond within 5s, independently from the completion of the command that has been requested by the Master.
    • The Master will repeat sending further HTTP POST requests and accepting acknowledgements until the Agent sends the final response, i.e. after completion of a task.
  • If the Agent does not receive a heartbeat from the Master within the double period (10s) then the Agent will assume the connection to be lost and will kill the task.
  • If the Master does not receive a heartbeat from the Agent then the Master will consider the task being lost and will assign the task an error state.
• Master/Agent Reconciliation
  
  • For a Network Connection Loss scenario the Master and Agent provide reconciliation capabilities:
    

    Jira
    server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JS-1524

    • The Agent will continue any running tasks up to the specified number of retry attempts to establish the communication by the Master.
      
      • Reconciliation will take place if the connection between Master and Agent can be established within the number of retries and if the Master has not been restarted.
      • Otherwise the Agent will assume the Master Service Failure scenario and will kill any running tasks.
        

        Jira
        server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JS-1523

         
    • This behavior applies to tasks that are executed by an Agent for a specific Master to which a connection has been lost. Tasks for other JobScheduler Master instances will be continued.
  • After a successful re-connect within the Network Connection Loss scenario the Master will repeat its request for execution of the respective jobs. Each new request includes an identifier for the previous execution request that allows the Agent to identify repeated requests:
    • for a job that has been completed within the time required to re-establish the connection the Agent will report the execution result back to the Master and will not re-execute the job.
    • for a job that is still running the Agent will report the appropriate information back to the Master which will note the running tasks and update JOC accordingly.
• Feature Availability
  

  • Display feature availability
    StartingFromRelease 1.10.2

...

...

• The currently supported measures include manual checking of Agent task logs after failure. 
  • The execution history of jobs that completed on an Agent during the Master Service Failure period is not reported back to the Master.
  • The Agent will kill running tasks after expiration of the Network Connection Loss scenario. Therefore it is recommended to check that the Agent tasks logs are checked for successful or unsuccessful execution of jobs.
• Automated recovery of the Master/Agent execution status after a Master Service Failure will be subject to future improvements.

  ...

  • • See

      Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JS-1549

      for more information.

  Change Management References

  Jira
  server SOS JIRA columns type,key,issuelinks,fixversions,status,priority,summary,updated maximumIssues 20 jqlQuery labels in (master-recovery) serverId 6dc67751-9d67-34cd-985b-194a8cdc9602

  ...

  Jira
  server SOS JIRA columns key,summary,type,createdkey,updatedissuelinks,duefixversions,assigneestatus,reporter,priority,statussummary,resolutionupdated maximumIssues 20 jqlQuery labels in (database-recovery) serverId 6dc67751-9d67-34cd-985b-194a8cdc9602

  ...