Introduction

• The JS7 offers to perform operations on Controllers and Agents by the JS7 - REST Web Service API.
  
  • For detailed information see the Technical Documentation of the REST Web Service API.
  • For available CLI operations see JS7 - Unix Shell Command Line Interface.
• The REST Web Service API can be accessed from Shell utilities such as curl.
• In addition, a PowerShell module is available for simplified access to the REST Web Service API. This is described in the JS7 - PowerShell Module article.

The Controller Status Operation Script offered for Unix Shell can be applied to perform frequently used status operations on Controllers and Agents.

Use of a Controller Cluster or Agent Cluster is subject to the JS7 - License.

Controller Status Operation Script

The script is offered for download and can be applied for frequently used status operations:

• The script is available for Linux and MacOS® using bash shell.
• The script terminates with exit code 0 to signal successful execution, with exit code 1 for command line argument errors and with exit code 4 for non-recoverable errors. Exit code 3 signals that no matching objects have been found.
• The script is intended as a baseline example for customization by JS7 users and by SOS within the scope of professional services. Examples make use of JS7 Release 2.7.2, bash 4.2, curl 7.29.0 and jq 1.6.0.
  

Prerequisites

The Script requires the curl utility and the jq utility to be available from the operating system. 

jq ships with the MIT license, see https://opensource.org/licenses/MIT.

Download

Download: JS7 - Download (Section: Unix Shell ClI)

Usage

Invoking the script without arguments displays the usage clause:

Usage: operate-controller.sh [Command] [Options] [Switches]

  Commands:
    terminate           --controller-id [--controller-url] [--switch-over]
    restart             --controller-id [--controller-url] [--switch-over]
    cancel              --controller-id [--controller-url]
    cancel-restart      --controller-id [--controller-url]
    switch-over         --controller-id
    appoint-nodes       --controller-id
    confirm-loss        --controller-id
    check               --controller-id --controller-url
    enable-agent        --controller-id --agent-id
    disable-agent       --controller-id --agent-id
    reset-agent         --controller-id --agent-id [--force]
    switch-over-agent   --controller-id --agent-id
    confirm-loss-agent  --controller-id --agent-id
    enable-subagent     --controller-id --subagent-id
    disable-subagent    --controller-id --subagent-id
    reset-subagent      --controller-id --subagent-id [--force]
    encrypt             --in [--infile --outfile] --cert [--java-home] [--java-lib]
    decrypt             --in [--infile --outfile] --key [--key-password] [--java-home] [--java-lib]

  Options:
    --url=<url>                        | required: JOC Cockpit URL
    --user=<account>                   | required: JOC Cockpit user account
    --password=<password>              | optional: JOC Cockpit password
    --ca-cert=<path>                   | optional: path to CA Certificate used for JOC Cockpit login
    --client-cert=<path>               | optional: path to Client Certificate used for login
    --client-key=<path>                | optional: path to Client Key used for login
    --timeout=<seconds>                | optional: timeout for request, default: 60
    --controller-id=<id>               | required: Controller ID
    --controller-url=<url>             | optional: Controller URL for connection test
    --agent-id=<id[,id]>               | optional: Agent IDs
    --subagent-id=<id[,id]>            | optional: Subagent ID
    --key=<path>                       | optional: path to private key file in PEM format
    --key-password=<password>          | optional: password for private key file
    --cert=<path>                      | optional: path to certificate file in PEM format
    --in=<string>                      | optional: input string for encryption/decryption
    --infile=<path>                    | optional: input file for encryption/decryption
    --outfile=<path>                   | optional: output file for encryption/decryption
    --java-home=<directory>            | optional: Java Home directory for encryption/decryption, default: $JAVA_HOME
    --java-lib=<directory>             | optional: Java library directory for encryption/decryption, default: ./lib
    --audit-message=<string>           | optional: audit log message
    --audit-time-spent=<number>        | optional: audit log time spent in minutes
    --audit-link=<url>                 | optional: audit log link
    --log-dir=<directory>              | optional: path to directory holding the script's log files

  Switches:
    -h | --help                        | displays usage
    -v | --verbose                     | displays verbose output, repeat to increase verbosity
    -p | --password                    | asks for password
    -k | --key-password                | asks for key password
    -o | --switch-over                 | switches over the active role to the standby instance
    -f | --force                       | forces reset on Agent
    --show-logs                        | shows log output if --log-dir is used
    --make-dirs                        | creates directories if they do not exist

see https://kb.sos-berlin.com/x/9YZvCQ

Commands

• terminate
  • Allows to terminate a Controller instance. If a Controller Cluster is used then no fail-over will occur as normal termination is not considered a failure situation.
  • When used with the active Controller instance in a Controller Cluster, users can apply the --switch-over switch to shift the active role on termination of the active cluster member. After restart the Controller instance will take the standby role in a Controller Cluster.
  • In a Controller Cluster the command will terminate the Controller instance identified by its URL using the --controller-url option.
• restart
  • Allows to restart a Controller instance. If a Controller Cluster is used then no fail-over will occur as normal termination is not considered a failure situation. The --switch-over switch can be used similarly to the terminate command.
  • In a Controller Cluster the command will restart the Controller instance identified by its URL using the --controller-url option.
• cancel
  
  • Allows to cancel a Controller instance. The Controller will immediately disconnect from Agents, will not create a journal snapshot and will terminate.
  • The command will cause fail-over in a Controller Cluster if used with the active Controller instance. After restart the Controller instance will take the standby role in a Controller Cluster.
  • In a Controller Cluster the command will cancel the Controller instance identified by its URL using the --controller-url option.
• cancel-restart
  • The command combines the operations to cancel and to restart a Controller instance.
• switch-over
  • The command allows to shift the active role in a Controller Cluster. For switch-over the Primary and Secondary Controller instance must be coupled.
• appoint-nodes

  • The command can be used in case that a Controller Cluster will not be coupled on initial operation.

  • The command is automatically sent by JOC Cockpit to Controller instances after restart.
• confirm-loss
  • The command can be used in a situation when the active JOC Cockpit Cluster Watch was not witness to failure of the active Controller Cluster member.
  • Users can confirm that the failed Controller Cluster member effectively is not running which allows the remaining Controller Cluster member to take the active role.
• check
  • Tests the connection between JOC Cockpit and a Controller instance.
  • The --controller-url option must be specified that will be used from JOC Cockpit to connect to the Controller. 
• enable-agent

  • Agents can be enabled after having been disabled. Enabled Agents are considered for job execution.

• disable-agent
  • When Agents are disabled, they are not considered for job execution. Running jobs can continue until completion.
• reset-agent

  • When an Agent is reset then the Agent will terminate and will restart. Job processes running in the Agent will be forcibly terminated and orders will be set to the failed state. When a forced reset is performed, then the operation forces an Agent to be reinitialized, to drop its journal and to be dedicated to the current Controller. Users are recommended to double-check if an Agent is not dedicated to a different Controller before using the --force switch.

• switch-over-agent

  • Allows to switch-over the active role between Director Agents in an Agent Cluster. For switch-over the Primary and Secondary Director instance must be coupled.

• confirm-loss-agent

  • The command can be used in a situation when the active Controller Cluster Watch was not witness to failure of the active Director Agent in an Agent Cluster. Users can confirm that the failed Director Agent effectively is not running which allows the remaining Director Agent member to take the active role.

• enable-subagent
  • Subagents can be enabled after having been disabled. Enabled Subgents are considered for job execution.
• disable-subagent
  • When Subgents are disabled, they are not considered for job execution. Running jobs can continue until completion. 
• reset-subagent
  • When a Subagent is reset then the Subagent will terminate and will restart. Job processes running in the Subagent will be forcibly terminated and orders will be set to the failed state. When a forced reset is performed, then the operation forces a Subagent to be reinitialized and to be dedicated to the current Agent Cluster. Users are recommended to double-check if a Subagent is not dedicated to a different Agent Cluster before using the --force switch.
• encrypt
  • Allows to encrypt a value using the --in option. If used to encrypt a file then --infile and --outfile options must be specified.
  • The --cert option specifies the path to a file holding the Certificate used for encryption.
  • Encryption is performed by Java libraries that are looked up in the ./lib sub-directory of the JS7 Unix Shell CLI. For details see JS7 - Encryption and Decryption.
• decrypt
  • Allows to decrypt a value using the --in option. If used to decrypt a file then --infile and --outfile options must be specified.
  • The --key option specifies the path to the Private Key used for decryption. If the Private Key is protected by a password, then the --key-password option or switch must be specified.
  • Decryption is performed by Java libraries that are looked up in the ./lib sub-directory of the JS7 Unix Shell CLI.

Options

• --url
  • Specifies the URL by which JOC Cockpit is accessible using <http|https>://<host>:<port>.
  • Example: http://centostest-primary.sos:4446
  • Example: https://centostest-primary.sos:4443
• --user
  • Specifies the user account for login to JOC Cockpit. If JS7 - Identity Services are available for Client authentication certificates that are specified with the --client-cert and --client-key options then their common name (CN) attribute has to match the user account.
  • If a user account is specified then a password can be specified using the --password option or interactive keyboard input can be prompted using the -p switch.
• --password
  • Specifies the password used for the account specified with the --user option for login to JOC Cockpit.
  • Password input from the command line is considered insecure.
    • Consider use of the -p switch offering a secure option for interactive keyboard input.
    • Consider use of the encrypt command to encrypt a password: ./deploy-workflow.sh encrypt --in=root --cert=encrypt.crt.
      • The encryption result will include the prefix enc: followed by the encrypted symmetric key, initialization vector and encrypted secret separated by space.
      • If an encrypted password is specified, then it will be decrypted using the Private Key file: ./deploy-workflow.sh <command> --password="enc:BF8J8KP7TPlxy..." --key=encrypt.key.
• --ca-cert
  • Specifies the path to a file in PEM format that holds the Root CA Certificate and optionally Intermediate CA Certificates to verify HTTPS connections to JOC Cockpit.
• --client-cert
  • Specifies the path to a file in PEM format that holds the Client Certificate if HTTPS mutual authentication is used..
• --client-key
  • Specifies the path to a file in PEM format that holds the Client Private Key if HTTPS mutual authentication is used..
• --timeout
  • Specifies the maximum duration for requests to the JS7 REST Web Service. Default: 60 seconds.
• --controller-id
  • Specifies the identification of the Controller.
• --controller-url
  • When used with the check command, specifies the protocol, host and optionally port of the Controller instance to which the connection is tested.
  • When using the terminate, restart, cancel and cancel-restart commands for a Controller Cluster, the Controller URL must be specified.
• --agent-id
  • The Agent ID specifies a unique identifier for a Standalone Agent or Agent Cluster. Agents are identified from their Agent ID.
  • When used with the enable-agent and disable-agent commands more than one Agent ID can be specified separated by comma.
• --subagent-id
  • The Subagent ID specifies a unique identifier for a Subagent in an Agent Cluster. Subagents are identified from their Subagent ID.
  • When used with the enable-subagent, disable-subagent and reset-subagent commands, the option specifies the related Subagent.
  • When used with the enable-subagent and disable-subagent commands more than one Subagent ID can be specified separated by comma.
• --key
  • When used with the decrypt command, specifies the path to a file that holds the Private Key used for decrypting in PEM format.
• --cert
  
  • When used with the encrypt command, specifies the path to a file that holds the CA signed or self-signed X.509 Certificate. Alternatively the path to a file holding the Public Key can be specified. The Certificate is expected in PEM format.
  • For encryption the Certificate must match the Private Key used for later decryption specified with the --key option.
• --key-password
  • When used with the decrypt command, specifies the password for access to the key file using the --key option.
  • Password input from the command line is considered insecure.
    • Consider use of the -k switch or more elaborate mechanisms, for example by temporarily populating the system keystore form a security key such as a YubiKey® or similar.
    • Consider use of encrypted passwords as explained with the --password option.
• --in
  • When used with the encrypt and decrypt commands, specifies the input value that should be encrypted or decrypted.,
  • One of the options --in or --infile can be specified.
• --infile
  • When used with the encrypt and decrypt commands, specifies the path to the input file that should be encrypted/decrypted.
  • One of the options --in or --infile can be specified. This option requires use of the --outfile option.
• --outfile
  • When used with the encrypt command, specifies the path to the output file that will be created holding the encrypted content of the input file.
  • When used with the decrypt command, specifies the path to the output file that will be created holding the decrypted content of the input file.
  • The option is required if the --infile option is specified
• --java-home
  • When used with the encrypt and decrypt commands or with encrypted passwords, specifies the Java home directory. By default the JAVA_HOME environment variable is used to determine the location of Java.
  • The Java home directory is the top-level directory of a Java installation. The directory includes the bin sub-directory and java executable.
• --java-lib
  • When used with the encrypt and decrypt commands or with encrypted passwords, a number of Java libraries are required to perform encryption/decryption.
  • The Java libraries are expected in the lib sub-directory of the JS7 Unix Shell CLI. Default: ./lib.
• --audit-message
  • Specifies a message that is made available to the Audit Log.
  • Specification of Audit Log messages can be enforced on a per user basis and for a JS7 environment.
• --audit-time-spent
  • Specifies the time spent to perform an operation which is added to the Audit Log.
  • The option can be specified if the --audit-message option is used.
• --audit-link
  • Specifies a link (URL) which is added to the Audit Log.
  • The option can be specified if the --audit-message option is used.
• --log-dir
  • If a log directory is specified then the script will log information about processing steps to a log file in this directory.
  • File names are created according to the pattern: deploy-workflow.<yyyy>-<MM>-<dd>T<hh>-<mm>-<ss>.log
  • For example: deploy-workflow.2022-03-19T20-50-45.log

Switches

• -h | --help
  • Displays usage.
• -v | --verbose
  • Displays verbose log output that includes requests and responses with the JS7 REST Web Service.
  • When used twice as with -v -v then curl verbose output will be displayed.
• -p | --password
  • Asks the user for interactive keyboard input of the password used for the account specified with the --user option..
  • The switch is used for secure interactive input as an alternative to use of the option --password=<password>.
• -k | --key-password
  • Asks the user for interactive keyboard input of the password used for access to a keystore or key file specified with the --keystore or --key options.
  • The switch is used for secure interactive input as an alternative to use of the --key-password=<password> option.
• -s | --switch-over
  • Specifies for terminate and restart commands to switch the active role in a Controller Cluster.
• -f | --force
  • When used with the reset-agent command for a Standalone Agent or Cluster Agent, and when used with the reset-subagent command for a Subagent, the option specifies that the Agent will terminate, will drop its journal and will restart. When resetting an Agent, job processes running in the Agent will be forcibly terminated and orders will be set to the failed state.
  • The operation forces an Agent to be reinitialized and to be dedicated to the current Controller or Agent Cluster in case of Subagents. Users are recommended to double-check that an Agent is not dedicated to a different Controller or Agent Cluster before using the switch.
• --show-logs
  • Displays log output created by the script if the --log-dir option is used.
• --make-dirs
  • If directories are missing that are indicated with the --log-dir option then they will be created.

Exit Codes

• 0: operation successful
• 1: argument errors
• 3: no objects found
• 4: JS7 REST Web Service is not reachable or reports errors

Examples

The following examples illustrate typical use cases for status operations on Controller and Agents.

Terminating, Restarting, Cancelling Controllers

Termination and restart of a Controller instance are offered by a number of commands.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# terminate Standalone Controller
./operate-controller.sh terminate "${request_options[@]}"

# restart Standalone Controller
./operate-controller.sh restart "${request_options[@]}"

# cancel Standalone Controller
./operate-controller.sh cancel "${request_options[@]}"

# cancel and restart Standalone Controller
./operate-controller.sh cancel-restart "${request_options[@]}"

When terminating/restarting a member in a Controller Cluster then --controller-url option must be used to specify which Controller instance should be terminated/restarted.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# terminate Controller Cluster instance
./operate-controller.sh terminate "${request_options[@]}" --controller-url=http://localhost:4444

# restart Controller Cluster instance
./operate-controller.sh restart "${request_options[@]}" --controller-url=http://localhost:4444

# cancel Controller Cluster instance
./operate-controller.sh cancel "${request_options[@]}" --controller-url=http://localhost:4444

# cancel and restart Controller Cluster instance
./operate-controller.sh cancel-restart "${request_options[@]}" --controller-url=http://localhost:4444

Switching-over, Appointing Nodes and Confirming Node Loss for Controller Cluster

Users can switch-over the active role in a Controller Cluster.

The appoint-nodes command is available in case that a Controller Cluster will not be coupled on initial operation.

The confirm-loss command can be used in a situation when the active JOC Cockpit Cluster Watch was not witness of failure of the active Controller Cluster member.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# switch-over active role in Controller Cluster
./operate-controller.sh switch-over "${request_options[@]}"

# terminate Controller instance and switch-over Controller Cluster
./operate-controller.sh terminate "${request_options[@]}" --controller-url=http://localhost:4444 --switch-over

# restart Controller instance and switch-over Controller Cluster
./operate-controller.sh restart "${request_options[@]}" --controller-url=http://localhost:4444 --switch-over


# appoint nodes for Controller Cluster
./operate-controller.sh appoint-nodes "${request_options[@]}"

# confirm node loss for Controller Cluster
./operate-controller.sh confirm-loss "${request_options[@]}" 

Enabling, Disabling, Resetting Agents

When Agents are disabled, they are not considered for job execution. 

When an Agent is reset then the Agent will terminate and will restart. Users are recommended to double-check if an Agent is not dedicated to a different Controller before using the --force switch.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# enable Standalone Agent
./operate-controller.sh enable-agent "${request_options[@]}" --agent-id=StandaloneAgent

# disable Standalone Agent
./operate-controller.sh disable-agent "${request_options[@]}" --agent-id=StandaloneAgent

# reset Standalone Agent
./operate-controller.sh reset-agent "${request_options[@]}" --agent-id=StandaloneAgent

# reset/force Standalone Agent
./operate-controller.sh reset-agent "${request_options[@]}" --agent-id=StandaloneAgent --force

For an Agent Cluster the reset-agent command is available. Enabling/disabling is performed at Subagent level.

When resetting an Agent Cluster then similar behavior applies as for Standalone Agents. Users should be aware that all Subagents in an Agent Cluster will be reset.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# reset Agent Cluster
./operate-controller.sh reset-agent "${request_options[@]}" --agent-id=AgentCluster

# reset/force Agent Cluster
./operate-controller.sh reset-agent "${request_options[@]}" --agent-id=AgentCluster --force

Switching-over and Confirming Node Loss for Agent Cluster

Users can switch-over the active role in an Agent Cluster.

The confirm-loss-agent command can be used in a situation when the active Controller Cluster Watch was not witness to failure of the active Director Agent in an Agent Cluster.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# switch-over active role in Agent Cluster
./operate-controller.sh switch-over-agent "${request_options[@]}" --agent-id=AgentCluster

# confirm node loss for Agent Cluster
./operate-controller.sh confirm-loss-agent "${request_options[@]}" --agent-id=AgentCluster

Enabling, Disabling and Resetting Subagents

When Subagents in an Agent Cluster are disabled, they are not considered for job execution. 

When a Subagent is reset then the Subagent will terminate and will restart. Users are recommended to double-check if a Subagent is not dedicated to a different Agent Cluster before using the --force switch.

# common options for connection to JS7 REST API
request_options=(--url=http://localhost:4446 --user=root --password=root --controller-id=controller)

# enable Subagent in Agent Cluster
./operate-controller.sh enable-subagent "${request_options[@]}" --subagent-id=Subagent_01

# disable Subagent in Agent Cluster
./operate-controller.sh disable-subagent "${request_options[@]}" --subagent-id=Subagent_01

# reset Subagent in Agent Cluster
./operate-controller.sh reset-subagent "${request_options[@]}" --subagent-id=Subagent_01

# reset/foce Subagent in Agent Cluster
./operate-controller.sh reset-subagent "${request_options[@]}" --subagent-id=Subagent_01 --force

Encrypting and Decrypting

• Secrets can be encrypted and decrypted as explaind with JS7 - Encryption and Decryption.
• Encryption/decryption require related Java libraries that are available from the ./lib sub-directory.
• For creation of a Private Key and Certificate see JS7 - How to create X.509 Encryption Keys.
  
  

# create Private Key
openssl ecparam -name secp384r1 -genkey -noout -out encrypt.key

# create Certificate Signing Request
openssl req -new -sha512 -nodes -key encrypt.key -out encrypt.csr -subj "/C=DE/ST=Berlin/L=Berlin/O=SOS/OU=IT/CN=Encrypt"

# create Certificate
openssl x509 -req -sha512 -days 1825 -signkey encrypt.key -in encrypt.csr -out encrypt.crt -extfile <(printf "keyUsage=critical,keyEncipherment,keyAgreement\n")


# encrypt a secret such as a password using the Certificate, the encryption result will be returned and will look like: enc:BEXbHYa...
 ./deploy-controller.sh encrypt --in="root" --cert=encrypt.crt

# options for connection to the JS7 REST API can specify the encryption result as password and the Private Key for decryption
request_options=(--url=http://localhost:4446 --user=root --password="enc:BEXbHYa..." --key=encrypt.key --controller-id=controller)

# for example, when restarting a Controller the Private Key is used to decrypt the password for access to the REST API on-the-fly
./operate-controller.sh restart "${request_options[@]}"

# decrypt an encrypted secret using the Private Key
./deploy-controller.sh decrypt --in="enc:BEXbHYa..." --key=encrypt.key

Resources

• API
  • JS7 - REST Web Service API
  • JS7 - PowerShell Module
• Controller Deployment
  • JS7 - Unix Shell CLI for Controller Deployment
• Workflow Deployment
  • JS7 - Unix Shell CLI for Workflow Deployment 
• Workflow Status Operations
  • JS7 - Unix Shell CLI for Workflow Status Operations