Agent Start-up

Start Script: agent.sh, agent.cmd

The JS7 Agent includes a Start Script for managing the Agent:

• The default location of the Start Script is:

  • /opt/sos-berlin.com/js7/agent/bin/agent.sh on Unix and

  • C:\Program Files\sos-berlin.com\js7\agent\bin\agent.cmd on Windows.
• Use of the Start Script is authoritative to start Agents.

Usage

Running the Agent Start Script without arguments displays the usage clause:

Usage: agent.sh command [options] [switches]
  command:
    start           [options]
    start-container [options]
    stop            [options]
    restart         [options]
    status          [options]
    cancel          [options]
    switch-over     [options]
    cert            [cert-options]         | see https://kb.sos-berlin.com/x/jLbAAw
  options:
    --http-port=<[interface:]port>         | http network interface and port, default: 4445
    --https-port=<[interface:]port>        | https network interface and port, default:
    --data-directory=<directory>           | default: /var/sos-berlin.com/js7/agent/var_4445
    --config-directory=<directory>         | default: /var/sos-berlin.com/js7/agent/var_4445/config
    --sigkill-delay=<seconds>              | send SIGTERM and delayed SIGKILL signal, default: 3
    --timeout=<seconds>                    | timeout for terminating jobs on Agent stop
    --java-options=<java options>          | default: -Xmx100m -Dfile.encoding=UTF-8; see https://kb.sos-berlin.com/x/uYo7B
  switches:
     -f | --fail-over                      | fail-over active role on stop and restart
     -c | --curl                           | use curl instead of Java http client
    --watchdog-log-stdout                  | use stdout channel instead of watchdog.log file

see https://kb.sos-berlin.com/x/ZqrAAw for more information.

Usage: agent.cmd command [options] [switches] 
  command:
    start           [options]
    stop            [options]
    restart         [options]
    status          [options]
    cancel          [options]
    switch-over     [options]
    start-service   [options]
    install-service [options]
    remove-service  [options]
    cert            [cert-options]         | see https://kb.sos-berlin.com/x/jLbAAw
  options:
    --http-port=<[interface:]port>         | http network interface and port, default: 4445
    --https-port=<[interface:]port>        | https network interface and port, default:
    --data-directory=<directory>           | default: /var/sos-berlin.com/js7/agent/var_4445
    --config-directory=<directory>         | default: /var/sos-berlin.com/js7/agent/var_4445/config
    --timeout=<seconds>                    | timeout for terminating jobs on Agent stop
    --java-options=<java options>          | default: -Xms100m -Dfile.encoding=UTF-8; see https://kb.sos-berlin.com/x/uYo7B
  switches:
    -f | --fail-over                       | fail-over active role on stop and restart
    -c | --curl                            | use curl instead of Java http client
    --watchdog-log-stdout                  | use stdout channel instead of watchdog.log file

see https://kb.sos-berlin.com/x/ZqrAAw for more information.

Command Line Options

• --http-port
  • Specifies the HTTP port that the Agent is listening to in order to receive requests from a JS7 Controller. This option can be used to indicate which network interface the JS7 Agent should listen to if a hostname or IP address is specified, for example with --http-port=myhost:4445.
  • Without this option being used the port defaults to 4445 and the Agent listens to all available network interfaces.
  • If a port is specified then the following order of precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable JS7_AGENT_HTTP_PORT (see below)
    • Third precedence: use of default value
• --https-port
  • Specifies the HTTPS port that the Agent is listening to in order to receive requests from a Controller. This option can be used to indicate which network interface the Agent should listen to if a hostname or IP address is specified, for example, with --https-port=myhost:4445.
  • When using the HTTPS protocol for connections from a Controller instance consider additionally allowing the HTTP protocol for local connections as with --http-port=localhost:4445. As the Agent Start Script makes use of an HTTP connection the protocol has to be in place to allow the Agent to be started, stopped, etc. by its Start Script.
  • Without this option being used the port defaults to 4445 and the Agent listens to all available network interfaces.
  • If a port is specified then the following order of precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable SJS7_AGENT_HTTPS_PORT (see below)
    • Third precedence: use of default value
• --data-directory
  • Specifies the location of the data directory that usually includes the config, logs, tmp and state sub-directories.
  • This location has to be unique for any Agent instance.
  • If a data directory is specified then the following order of precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable JS7_AGENT_DATA (see below)
    • Third precedence: use of default value JS7_AGENT_HOME/var_<JS7_AGENT_PORT>
• --config-directory
  • Specifies the location of the config directory for configuration data.
  • This location has to be unique for any Agent instance.
  • Should a configuration directory be specified then the following order of precedence applies:
    • First precedence: command line option
    • Second precedence: environment variable JS7_AGENT_CONFIG_DIR (see below)
    • Third precedence: use of default value JS7_AGENT_HOME/var_<JS7_AGENT_PORT>
• --sigkill-delay
  • Specifies the delay for terminating job processes on Unix in case the Agent is terminated. A value --sigkill-delay=5 will specify 5 seconds. Default: 3.
  • When the Agent is crashed or is stopped then remaining job processes and child processes will be forcibly terminated by the JS7 - Agent Watchdog.
    • if the --sigkill-delay option specifies a positive, non-zero value
      • send processes and child processes created by the Agent the SIGTERM signal,
      • wait for the period specified with the --sigkill-delay option,
    • send remaining job processes and child processes the SIGKILL signal.
  • When using JS7 - systemd Service Files for automated Startup and Shutdown with Unix Systems users should consider that --timeout and --sigkill-delay values will count towards the systemd service file's  TimeoutStopSec setting. Users must ensure that the Agent is terminated when the systemd timeout is exceeded.
  • The option is available for Agents running on Unix starting from release 2.7.2, see JS-2148 - Getting issue details... STATUS
• --timeout

  • With the stop and restart commands this option waits for the indicated duration and terminates the Agent including job processes if the timeout is exceeded.

  • The option accepts a number of seconds or the never value to wait endlessly for jobs to terminate.
  • When using JS7 - systemd Service Files for automated Startup and Shutdown with Unix Systems users should consider that --timeout and --sigkill-delay values will count towards the systemd service file's  TimeoutStopSec setting. Users must ensure that the Agent is terminated when the systemd timeout is exceeded.
• --kill-script

  • The kill script provides the functionality to kill tasks and child processes of jobs.

  • Kill scripts are by default provided from the following locations:
    

    • JS7_AGENT_DATA/work/kill_task.sh for Unix.

    • JS7_AGENT_DATA\work\kill_task.cmd for Windows.

  • The option can be used to specify the location of an individual kill script if required.

  • Should an individual kill script be specified then the following order of precedence applies:

    • First precedence: command line option
    • Second precedence: environment variable JS7_AGENT_KILL_SCRIPT (see below)
    • Third precedence: use of default value
  • Starting from release 2.7.2 the option is ignored, see JS-2148 - Getting issue details... STATUS
• --java-options

  • This option can be used to apply Java options for the Agent, e.g. for memory settings, see JS7 - FAQ - Which Java Options are recommended.

  • Without this option being used the Java options default to -Xms100m.

  • In order to specify a number of Java options quotes have to be used like this:
    • --java-options="-Xms100m -Xmx1g"

  • When specifying Java options then the following order of precedence applies:

    • First precedence: command line option
    • Second precedence: environment variable JAVA_OPTIONS (see below)
    • Third precedence: use of default value

Switches

• -c ,  --curl
  • Specifies that the curl utility should be used instead of the built-in HTTP client when sending commands to the Agent.
• -f, --fail-over

  • With the stop and restart commands this option will fail-over the active role to the Standby Director Agent instance in an Agent Cluster.

  • The switch is available starting from release 2.7.3. Prior to release 2.7.2 fail-over is automatically applied, starting from release fail-over requires the switch.
• --sigkill
  • Starting from release 2.7.2 the switch is ignored.
• --watchdog-log-stdout
  • Specifies that log output to stdout and stderr channels will not be redirected to a file. By default log output to stdout and stderr channels is redirected to the watchdog.log file.
  • The switch is available starting from release 2.7.2.

Watchdog Script agent_watchdog.sh, agent_watchdog.cmd

Technically the Agent is started and is restarted from its Watchdog Script, for details see JS7 - Agent Watchdog.

The Watchdog Script is used to terminate job processes and child processes in case that the Agent is crashed.

Instance Start Script: agent_<port>.sh, agent_<port>.cmd

The JS7 Agent includes a template file for an Instance Start Script that can be adjusted to specify start-up parameters.

• The Instance Start Script sets a number of environment variables and finally executes the Start Script.
• In the Instance Start Script start-up parameters can be specified from environment variables, see below.

It is recommended that an individual Instance Start Script is created from the template file as this allows to specify individual start-up parameters from environment variables.

Location

• The default location of the Instance Start Script template file is:

  • /opt/sos-berlin.com/js7/agent/bin/agent_instance.sh-example on Unix systems and

  • C:\Program Files\sos-berlin.com\js7\agent\bin\agent_instance.cmd-example on Windows.
• Users can create a copy of the template file and remove the -example extension from the file name. The renamed Instance Start Script will not be overwritten when updating the Agent later on.
• It is recommended that the template file is copied to an Instance Start Script file with the name agent_<port>.sh|.cmd with <port> being the HTTP port used by the Agent (see below).
  • If the Agent installation is performed on Windows using the installer, either in headless mode or in graphical mode, then the installer will automatically create this script.
  • If the Agent installation is performed by extracting a .tar.gz/.zip archive then the Instance Start Script has to be created from a copy of the template file.
  • Users can adjust start-up parameters from environment variables once the Instance Start Script has been copied, as described below.

Usage

The Instance Start Script can be executed with the same arguments as the Start Script, as described above.

Agent Environment Variables

Most environment variables in the Instance Start Script correspond to command line options in the Start Script (described above). The following environment variables can be modified in the script:

• JS7_AGENT_HOME
  
  • Points to the directory where the JS7 Agent has been installed.
  • If this environment variable is not set then the default value will be used, which is the parent directory of the Agent Start Script.
  • If the Agent is started from a directory other than the Agent installation directory, for example by copying the Start Script to some other location, then this environment variable has to be set in order to locate the JS7 Agent installation directory.
  • There is no corresponding command line option.
• JS7_AGENT_USER
  • Sets the user account that the Agent is operated for. This includes running jobs with the permissions of the specified user account.
  • This setting is available for Unix systems only. For Windows systems the user account that runs the Windows Service is used.
  • Without setting this environment variable the user account that runs the Start Script is used.
  • This setting can be used when running the Agent Start Script on system start-up and shutdown that are executed by root, for example in /etc/init.d and corresponding locations or with systemd.
  • There is no corresponding command line option.
• JS7_AGENT_HTTP_PORT
  • Sets the HTTP port that the Agent is listening to.
  • Indicates which network interfaces the Agent should listen to if a hostname or IP address is specified, for example with JS7_AGENT_HTTP_PORT=myHost:4444.
  • If only a port number is specified then the Agent will listen to all available network interfaces via HTTP.
  • Without setting this environment variable the default port of 4445 will be used.
  • Corresponding command line option: --http-port
• JS7_AGENT_HTTPS_PORT
  • Sets the HTTPS port that the Agent is listening to.
  • Indicates which network interfaces the Agent should listen to if a hostname or IP address is specified, for example with JS7_AGENT_HTTPS_PORT=myHost:4443.
  • If a port number only is specified then the Agent listens to all available network interfaces via HTTPS.
  • When using the HTTPS protocol for connections from a Controller users should consider to allow the HTTP protocol for local connections as with JS7_AGENT_HTTP_PORT=localhost:4445. As the Agent Start Script makes use of an HTTP connection, this protocol has to be in place to allow the Agent instance to be started, stopped etc. by its Start Script.
  • Without setting this environment variable the HTTPS protocol is not used.
  • Corresponding command line option: --https-port
• JS7_AGENT_DATA
  • Points to the location where the Agent finds its configuration data, log data and journals.
  • Without setting this environment variable the default value is JS7_AGENT_HOME/var_<JS7_AGENT_HTTP_PORT>.
  • Corresponding command line option: --data-directory
• JS7_AGENT_CONFIG_DIR
  • Points to the directory where the Agent finds its configuration data.
  • Without setting this environment variable the default value is JS7_AGENT_DATA/config.
  • Corresponding command line option: --config-directory
• JS7_AGENT_LOG_DIR
  • Specifies the directory where the Agent log files are created.
  • This setting defaults to the sub-directory logs in the JS7_AGENT_DATA directory.
  • There is no corresponding command line option.
• JS7_AGENT_PID_FILE_DIR
  • Specifies the directory where the Agent PID file is created.
  • This setting defaults to the directory that is specified with the JS7_AGENT_LOG_DIR environment variable or the log directory default value.
  • There is no corresponding command line option.
• JS7_AGENT_PID_FILE_NAME
  • Sets the file name to which the JS7 Agent stores its PID.
  • The default value is agent.pid.
  • There is no corresponding command line option.
  • FEATURE AVAILABILITY STARTING FROM RELEASE 2.5.1
• JS7_AGENT_WORK_DIR
  • Specifies the working directory for the jobs started by the Agent, e.g. ${HOME} or %USERPROFILE%.
  • This setting defaults to JS7_AGENT_DATA.
  • There is no corresponding command line option.
• JS7_AGENT_KILL_SCRIPT
  • Specifies the location of a kill script if an individual script is required.

  • The kill script allows the task and any child processes of a job to be killed.

  • Kill scripts are by default provided at the following locations:
    

    • JS7_AGENT_DATA/tmp/kill_task.sh for Unix.

    • JS7_AGENT_DATA\tmp\kill_task.cmd for Windows.

  • Corresponding command line option: --kill-script
  • Starting from release 2.7.2 this environment variable is ignored, see JS-2148 - Getting issue details... STATUS
• JS7_AGENT_TZ
  • Specifies the Agent's time zone that is used for example for timestamps in log output.
  • For the list of available TZ time zone values see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
• JAVA_HOME
  • Points to the location of the JVM, either a JRE (Java Runtime Environment) or JDK (Java Development Kit).
  • When this environment variable is not set, Java will be used from the location specified by the system path.
  • The JAVA_HOME environment variable does not necessarily point to the location of a JDK but to a JRE directory where the bin/java executable resides. For example, if the location of the Java executable is /opt/java/jdk8u202-b08/jre/bin/java then use JAVA_HOME=/opt/java/jdk8u202-b08/jre. If the location is /opt/java/jdk-21/bin/java then use JAVA_HOME=/opt/java/jdk-21.
  • There is no corresponding command line option.
• JAVA_OPTIONS
  • Sets Java options, e.g. the Java memory settings for the Agent.
  • When this environment variable is not set, the Java options will default to -Xms100m.
  • Corresponding command line option: --java-options

Individual Environment Variables

In addition to Agent Environment Variables individual environment variables can be added. Such environment variables are made available for jobs like this:

SCRIPT_PATH=/var/scripts
export SCRIPT_PATH

set SCRIPT_PATH=C:\ProgramData\scripts

Running the Agent

• SOS does not recommend running the JS7 Agent with the root account (for Unix) or the Administrator account (for Windows).
  • In fact the Agent can be operated with administrative privileges which leverages switching to any user account without specifying a password if jobs are to be executed for a different user account.
  • However, in compliance-aware environments this is considered an unwanted risk and in any other environments this simply is a bad idea.
• For Unix use of sudo is recommended.
• For Windows the JS7 Agent can switch to a different user account in a compliant way, see JS7 - Running Jobs as a different User.

The examples listed below make use of an Instance Start Script agent_4445.sh|.cmd, which is created from the template file to run the Agent using port 4445.

Starting the Agent

agent_4445.sh|.cmd start [options]

When used for Container operation using Unix OS then the following command is used:

agent_4445.sh start-container [options]

The command is used to start the JS7 Agent in a container:

• The command will keep the start script running as PID 1 in order to receive later SIGTERM, SIGKILL signals from the container management software.
• The command start-docker is an alias for start-container.
• SET-223 - Getting issue details... STATUS

Stopping the Agent

# terminate jobs and stop Agent
agent_4445.sh|.cmd stop [options]

# terminate jobs on Unix by sending SIGKILL with a delay of 10s and stop Agent
agent_4445.sh stop --sigkill-delay=10

# wait for jobs to terminate within 60s and stop Agent
agent_4445.sh|.cmd stop --timeout=60

# wait for jobs to terminate and stop Agent
agent_4445.sh|.cmd stop --timeout=never

# stop active Director Agent instance and fail-over active role
agent_4445.sh|.cmd stop --fail-over

This command will terminate the Agent.

• Unix
  • Job processes and child processes will be terminated and the Agent will terminate with exit code 0.
    • collect PIDs of job processes,
    • send SIGTERM signal to remaining job processes and child processes for which PIDs have been collected,
    • wait for the grace timeout specified with the job or for stdout/stderr to be released by processes whichever is earlier,
    • send SIGKILL signal to job processes and child processes,
    • terminate the Agent with exit code 0.
• Windows
  • Job processes and child processes will be forcibly terminated and the Agent will terminate with exit code 0.
• Using the --timeout option will wait for the indicated timeout that the Agent terminates. If the timeout is exceeded and if the Agent is still running, then Agent and jobs will be focibly terminated. The option accepts a number of seconds or the never value that indicates that the Agent will wait endlessly until job processes and child processes will terminate.
  
  • Agent
    • collect PIDs of job processes,
    • terminate the Agent with exit code 143 for Unix and 99 for Windows.
  • Watchdog
    • Unix
      • if the --sigkill-delay option is used
        • send SIGTERM signal to remaining job processes and child processes for which PIDs have been collected,
        • wait for the indicated delay or for stdout/stderr to be released by processes whichever is earlier,
      • send SIGKILL signal to remaining job processes and child processes.
    • Windows
      • forcibly terminate job processes and child processes.
• The --fail-over option can be used when stopping a Director Agent that holds the active role. The switch will force fail-over to the standby Director Agent. Without using this switch no fail-over will occur and the Agent Cluster will not be functional after the active Director Agent is stopped. The switch is introduced with release 2.7.3.

agent_4445.sh|.cmd abort [options]

The Agent process will terminate.

Starting from release 2.7.2 the command is deprecated and is replaced by the stop command, optionally for Unix using the --sigkill-delay=0 option.

agent_4445.sh|.cmd cancel | kill [options]

The Agent process is forcibly terminated.

• The kill command can be used as an alias to cancel.
• The command corresponds to sending a SIGKILL signal with a kill -9 OS command (Unix) or taskkill command (Windows) to the Agent.
• Should jobs be running then they will be terminated by the JS7 - Agent Watchdog similar to crash of the Agent.

Restarting the Agent

# terminate jobs and restart Agent
agent_4445.sh|.cmd restart [options]

# terminate jobs on Unix by sending SIGKILL with a delay of 10s and restart Agent
agent_4445.sh restart --sigkill-delay=10

# wait for jobs to terminate and restart Agent
agent_4445.sh|.cmd restart --timeout=never

# restart active Director Agent instance and fail-over active role
agent_4445.sh|.cmd restart --fail-over

The command will restart the Agent.

• Job processes and child processes will be terminated and the Agent will terminate as explained in chapter Stopping the Agent.
• The Agent is started similar to use of the start command using the same options as for initial start.
• The --timeout option for Unix/Windows and the --sigkill-delay option for Unix can be used.

Checking the Agent Status

agent_4445.sh|.cmd status [options]

If the Agent is up and running then this command results in output such as:

version: 2.0.0-alpha.20210722.2 (2021-07-22)
buildId: QQqWYNiJRbqcYqx4iiFWww
startedAt: 1626981330629
isTerminating: false
system:
  hostname: agent-2-0-primary
  distribution: Alpine Linux v3.13
  cpuModel: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  mxBeans:
    operatingSystem:
      availableProcessors: 8
      systemLoadAverage: 0.0107421875
java:
  version: 1.8.0_292
  memory:
    maximum: 954728448
    total: 289931264
    free: 87118936
  systemProperties:
    java.vendor: IcedTea
    os.arch: amd64
    java.runtime.name: OpenJDK Runtime Environment
    os.version: 3.10.0-957.1.3.el7.x86_64
    os.name: Linux
    java.vm.name: OpenJDK 64-Bit Server VM
    java.version: 1.8.0_292

Switching-over the Active Director Agent Instance

agent_4445.sh|.cmd switch-over [options]

If an Agent Cluster is operated then the command will switch the active role from the Active Director Agent instance to the Standby Director Agent instance. The command can be executed for the Active Director Agent .

After switch-over the Agent instance is restarted.

Performing Certificate Rollout

agent_4445.sh|.cmd cert [cert-options]

This command is used to generate an SSL/TLS Certificate for client and server authentication. 

For details and command line options see JS7 - Certificate Authority - Rollout Certificates for HTTPS Connections

Windows Service Interface

The following information applies to headless installation for Windows systems. For installation with a GUI and user dialog see JS7 - Agent - Installation Using the Windows Graphical Installer.

The Agent is operable as a Windows Service. The Start Script of the Agent allows the installation/removal of the Windows Service.

The examples below make use of an Instance Start Script agent_4445.cmd, which is created from the template file to run the Agent for port 4445.

Installing the Windows Service

agent_4445.cmd install-service [--http-port=<[interface:]number>]

The command installs the Agent's Windows Service. After the installation, users find the Windows Service with the name SOS JS7 Agent -port=<number> in the Services Manager Console. The Windows service uses the local system account.

Starting the Windows Service

agent_4445.cmd start-service [--http-port=<[interface:]number>]

The command starts the Windows Service with the name SOS JS7 Agent -port=<number>.

Removing the Windows Service

agent_4445.cmd remove-service [--http-port=<number>]

The command removes the Windows Service. After executing the command users should not find the Windows Service with the name SOS JS7 Agent -port=<number> in the Services Manager Console any longer.

Configuring the Windows Service

After installation of the Windows Service users find the .\service\js7_agent_<http-port>w.exe file. The program can be executed to configure the Windows Service.

Debugging the Windows Service

• For general information see JS7 - Log Levels and Debug Options. 
• The log level can be increased by using the Agent's Apache ProcRun Daemon Service Manager daemon/service. On Windows systems this is installed in the Agent's service directory with a name such as js7_agent_<port>w.exe  where <port> is the Agent's HTTP port.
• Users can start the ProcRun Manager, select the Logging tab in the user interface and set the log level to Debug. 
• For the location of log files see JS7 - Log Files and Locations.
• Users should consider reverting the debug log level to its original value once they are finished with debugging.

Resources

• JS7 - FAQ - How does JobScheduler terminate Jobs
• JS7 - FAQ - Can JobScheduler restart failed Jobs