Page History

...

• The wording of cluster terms suggests to indicate the Active Controller Instance and the Standby Controller Instance independently from the fact if the Primary or Secondary Controller Instance is active.
• A Controller implements an active-passive cluster, however, the term passive is misleading as the Standby Controller Instance is not passive at all but records any order state transitions occurring in the Active Controller Instance. Both Controller instances hold a journal of order state transitions that is actively synchronized. Fail-over and switch-over will occur only if both Controller instances' journals are in sync.
• The Cluster presents itself as a single unit to the outside world, i.e. to JOC Cockpit and to Agents.
  • Any operations performed in JOC Cockpit are automatically applied to the Active Controller Instance.
  • At any point in time only one Controller instance is active and the other instance is in standby mode.

...

On start-up both Primary and Secondary Controller instances connect establish a connection to the Cluster Watch Agent.

...

• The Cluster Watch Agent knows immediately when the Active Controller Instance is down due to a connection loss from this instance.
• The Standby Controller Instance holds a connection to the Active Controller Instance and knows immediately when this connection is lost.
• Failure of the Active Controller Instance is the point in time when the Standby Controller Instance and the Cluster Watch Agent check to find common ground about a cluster fail-over operation: They determine if they should declare the Active Controller Instance being inoperable and after a short period of 1-2s they proceed and cast their votes if the Standby Controller Instance should now become the Active Controller Instance.
• As a prerequisite for fail-over both the Cluster Watch Agent and the Standby Controller Instance have to confirm that the Standby Controller Instance's journal was in sync with the Active Controller Instance at the point in time of failure.

...

• If the Cluster Watch Agent is terminated at the same time as a failed Active Controller Instance then no fail-over can occur.
• If the Cluster Watch Agent is terminated at the same time as one of the Controller instances then the Controller Cluster cannot start up as this requires operational readiness of the Cluster Watch Agent.
• A Cluster Watch Agent that is started after failure of the Active Controller Instance is disqualified from casting its vote as it has no knowledge if the Controller instances' journals are  in sync.

...

The best advice is not to apply such automated clustering mechanisms, but to perform manual fail-over. Reasons include but are not limited to the following issues:

• The cluster has to guarantee that only one of both Standalone Controller instances is started at any point in time.
  • Should this rule not be observed then both Controller instances will request Agents to execute the same workflows and jobs which will result in double job execution.
  • Controller journals will be messed up with the same orders in different state transitions.
  • In this situation the only solution is to drop both Controller instances' journals that are available from the state sub-directory, to accept that any orders are lost and to redeploy scheduling objects.
• There is no simple way to determine if a Controller instance is not in perfect shape condition to manage orders.
  • Performing PID file checks is of limited use: this can prove the unavailability of a Controller instance. However, a positive PID file check does not prove that a Controller instance works.
  • Log file analysis is pointless. Controllers are heavily making use of asynchronous operations when it comes to Agents. Occurrence of error messages in log files includes to allow a situation to be recovered within the next few seconds.
• A Controller Cluster guarantees high availability when used with a JS7 - Agent Cluster. Use of Standalone Agents limits high availability.

...