JS7 JobScheduler

Space shortcuts

Page tree

Versions Compared

Old Version 14

changes.mady.by.user Andreas Püschel

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Püschel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Introduction

For JS7 - Automated Deployment On Premises use of a Deployment Server Area is recommended.

The Deployment Server Area is used for the following purposes:

  • to hold the script environment for JS7 - Automated Installation and Update and the JS7 - Deployment Packaging Script,
  • to hold the configuration files and optionally certificates for deployment of JS7 componentsproducts,
  • to hold the JS7 installations installation per JS7 release and component product such as JOC Cockpit, Controller, Agent,
  • to hold the archive of deployment packages Deployment Packages per JS7 releases and target machinehost,
  • to operate a JS7 Agent that is used to perform the JS7 - Deployment Workflow.

Purpose

Use of a Deployment Server Area is beneficial for the following purposes:

  • Desired State Configuration
    • The JS7 - Deployment Descriptor specifies the desired state of the installation and configuration on a number of target machineshosts.
    • The concept includes not to deploy individual files or configuration items but to view an installation holistically with all installation files and configuration files being deployed to guarantee consistency.
  • Provisioning
    • Considering Desired State Configuration it makes no difference if a JS7 component is product will be initially installed, is will be updated, upgraded or patched.
    • Deployment processes are repeatable at any time. This suggests not to rely on backups of target machines hosts for the purpose of restoring JS7 components in products in case of data loss in the target machine. Instead, deployment is repeated in case that a target machine host has to be newly set up from scratch. Consider which data will be lost in case that a target machinehost's file system will crash:
      • Log Data: retention of older JS7 component product log files might not be too relevant your primary concern in a crash situation. At the same time most recent additions to log files might anyway be lost. Consider that log data exclude job logs and order logs that are streamed in near real-time from Agents via Controllers Controller instances to the JOC Cockpit database.
      • Transaction Data: due to the ephemeral nature of JS7 - Order State Transitions there is no way to in consistently backup backing up a Controller's or Agent's journal. Instead, clustering of Controllers Controller and Agents Agent instances on separate machines hosts brings the required redundancy.
      • For details see JS7 - How to backup the job scheduling environment.
  • Probing
    • The deployment process includes to temporarily install JS7 components in the working area products in a temporary work directory of the Deployment ServerArea. 
    • This allows users to check if all required configuration files including certificates are in place.
    • In most situations it will be possible to start up the JS7 components on products in the Deployment ServerArea. This might not include for example to run jobs with a JS7 Agent as the job environment might not be available with the Deployment ServerArea. However, it allows to check if the JS7 component product will perfectly start - using the expected configuration items such as certificates - and will terminate normally. In addition it includes to check the JS7 componentproduct's log files for warnings and errors on start-up and termination.
  •  Parallelism
    • During upgrade to newer minor JS7 releases users might find an inconsistent status of JS7 componentsproducts for a short period of time.
      • This applies to upgrading, for example from a 2.2.1 release to a 2.4.1 release. 
      • This does not apply to updating, for example from a 2.2.1 release to a 2.2.3 release.
    • A newer Controller release 2.5.1 might not work with an older Agent release 2.2.3. As the Controller has to be upgraded before Agents are upgraded users might find a period during which no jobs are executed on Agents using older releases. In fact the related orders will not be lost, however, they will be put to the BLOCKED state until the Agent is upgraded. It is therefore preferable to have the duration for upgrading a larger number of Agents as short as possible.
    • The JS7 - Deployment Workflow provides unlimited parallelism for deployments as it can run thousands of tasks in parallel on the same Agent.
      • Typically the time to shut down a JS7 Agent, to replace its installation and to restart the JS7 Agent requires less than 60s.
      • When stopping Agents Agent instances during the deployment procedure this will allow an individual Agent to delay termination until any running tasks are completed. This includes to delay the update/upgrade process. At the same time this guarantees that no running tasks are killed. However, users can force immediate termination of Agents during deployment, see JS7 - Agent Command Line Operation.
  • Fallback
    • Though use of a Deployment Server Area allows probing to some extent there might be situations when deployment to a target machine host fails. Such situations frequently do not suggest lengthy analysis of problems with the target machine host but require immediate action to roll back the installation and to fall back to the a previous release.
    • With Deployment Packages and Deployment Descriptors for previous releases being available from the Deployment Server Area users can initiate the fallback procedure to deploy a previous release within seconds.
    • Error handling in Deployment Workflow Workflows allows to decide if deployments are considered being failed for example with a single Agent not being successfully deployed or if user intervention is required to decide if processing of the Deployment Workflow should be cancelled or if execution for individual target machines is hosts should be retried.

Prerequisites

No specific software requirements apply to a Deployment Server. The following factors should be considered:

  • Storage
    • Storage capacity has to match the number of deployment packages that are stored per JS7 release, per JS7 component and target machines.
      • Example
        • Assume that the Deployment Server is used to deploy JS7 Agents to 100 target machines.
        • The average disk space consumption of a JS7 Agent package is around 175 MB.
        • If users intend to keep the latest three JS7 releases then this will make 300 packages with an overall disk space consumption of approx. 53 GB.
    • The working area in the Deployment Server is used to temporarily store JS7 installations. A rough estimate of 10 GB will be sufficient for most purposes.
    • Storage for JS7 release tarballs should be considered. For Unix environments the tarballs for JOC Cockpit, Controller and Agent require approx. 600 MB per release.
  • Computing Power
    • Speed matters only when it comes to packaging. This process is performed once per new JS7 release that should be deployed.
    • It can be considered a waste of resources having a powerful server being used <1% of the time.
  • Network
    • The Deployment Server requires decent connectivity as it has to connect to any target machine to which JS7 components should be deployed.
      • All network connections are from the Deployment Server to the target machine, not vice versa.
      • Considering firewalls users might think about "opening the gates" for the timeslot of deployment only.
    • Transfer of tarball packages to target machines preferably happens in parallel to have a short period of time in which mixed releases of JS7 components are in place.

Security

It is essential to have a consistent and secure deployment process in place.

  • Security is the basic reason why the JS7 deployment scripts ship as shell scripts and not from a high level programming language that might require installation and updates. Transparency and adjustability are key. Users are free to modify deployment scripts at their will and to request changes within the scope of SOS professional services.
  • The source of JS7 installation tarballs is a major concern. Basically the installer scripts for JS7 components offer to download components on demand from the SOS Web Site. Users are discouraged to make use of this feature for automated deployment. Instead, the JS7 components should be separately downloaded and checked that no tampered versions of the files have been download to exclude man-in-the-middle attacks.
  • Good understanding of the deployment process and of the working of deployment scripts is key.

...

The suggested directory layout for the Deployment Server includes the following directories and files:

...

  • js7_deploy_agent_unix.<agent-id>.<release>.config.tar.gz (deployment package for the Agent's configuration directory)
  • js7_deploy_agent_unix.<agent-id>.<release>.install.tar.gz (deployment package for the Agent's installation directory)
  • run_deploy_agent.sh (deployment script)
  • run_install_agent.sh (wrapper script for the parameterized call to js7_install_agent.sh)

...

  • <controller-id> (specifies the Controller ID)
    • <controller-type> (specifies the type of the Controller instance which is primary or secondary) 
      • js7_deploy_controller_unix.<controller-id>.<controller-type>.<release>.config.tar.gz (deployment package for the Controller instance's configuration directory)
      • js7_deploy_controller_unix.<controller-id>.<controller-type>.<release>.install.tar.gz (deployment package for the Controller instance's installation directory)
      • run_deploy_controller.sh (deployment script)
      • run_install_controller.sh (wrapper script for the parameterized call to js7_install_controller.sh)
    • <controller-type> (specifies the type of the Controller instance which is primary or secondary) 

...

...

  • certs (holds CA-signed Certificates)
  • csr (holds Certificate Signing Requests)
  • private (holds Private Keys)

...

The JS7 - Deployment Packaging requires the jq utility to be available from the operating system. 

Version 1.6 of jq is available from the js7_deploy.tar.gz archive, see Download section, and ships with the MIT license, see https://opensource.org/licenses/MIT.

No additional software requirements apply to a Deployment Area. The following resources should be considered:

  • Storage
    • Storage capacity has to match the number of Deployment Packages that are stored per JS7 release, per JS7 product and target host.
      • Example
        • Assume that the Deployment Area is used to deploy JS7 Agents to 100 target hosts.
        • The average disk space consumption of a JS7 Agent package is around 175 MB.
        • If users intend to keep the latest three JS7 releases then this will make 300 packages with an overall disk space consumption of approx. 53 GB.
    • The work directory in the Deployment Area is used to temporarily store JS7 installations. A rough estimate of 10 GB will be sufficient for most purposes.
    • Storage for JS7 release installation files should be considered. For Unix environments the tarballs for JOC Cockpit, Controller and Agent require approx. 600 MB per release.
  • Computing Power
    • Speed matters when it comes to packaging. However, this process is performed only once per new JS7 release that should be deployed.
    • It can be considered a waste of resources having a powerful server being used <1% of the time as computing power is required for the packaging step only. Users should identify a way how to allocate computing resources exclusively in case that they are needed.
  • Network
    • The host operating the Deployment Area requires decent connectivity as it has to connect to any target hosts to which JS7 products should be deployed.
      • Network connections are established from the Deployment Area's server to the target hosts, not vice versa.
      • Considering firewalls users might think about "opening the gates" for the timeslot of deployment only.
    • Transfer of tarball packages to target hosts preferably happens in parallel to have a short period of time in which mixed releases of JS7 products are in place.

Security

It is essential to have a consistent and secure deployment process in place.

  • Security is the basic reason why the JS7 deployment scripts ship as shell scripts and not from a high level programming language that might require installation and updates. Transparency and adjustability are key. Users are free to modify deployment scripts at their will and to request changes within the scope of SOS professional services.
  • The source of JS7 installation tarballs is a major concern. Basically the installer scripts for JS7 products offer to download products from the SOS Web Site on demand. Users are discouraged to make use of this feature for automated deployment. Instead, the JS7 products should be separately downloaded and checked that no tampered versions of the files have been downloaded.
  • Good understanding of the deployment process and of the working of deployment scripts is key for security.

Anchor
download
download
Download

A tarball of the directories and scripts that make up the Deployment Area is provided per JS7 release. 

For download refer to JS7 - Download and select the js7_deploy.tar.gz tarball.

Download can be performed using curl for example with the following URL (users should consider to download from the latest release):

Code Block
languagebash
titleDownload Deployment Area tarball from SOS Web Site
curl "https://download.sos-berlin.com/JobScheduler.2.5/js7_deploy.tar.gz" --output "js7_deploy.tar.gz"

Deployment Steps

Deployment is performed by the following steps:

  • Create the JS7 - Deployment Descriptor from a .json file.
  • Run the JS7 - Deployment Packaging script indicating the .json Deployment Descriptor file. This step will create related Deployment Packages in the <archive>/<deployment-descriptor> directory.
  • Perform deployment by applying one of the following strategies:
    • Run the Deployment Script created from the previous step.
      • The script is available from the <archive>/<deployment-descriptor>/run_deploy.sh file and can be executed from the command line

...

  • config (general configuration)
    • private (specific configuration)
      • trusted-pgp-keys (optionally holds PGP public key files and keyring files used for signing, see JS7 - Deployment of Scheduling Objects)
        • <pgp-public-key> (public key file or keyring file)
        • <pgp-public-key> (public key file or keyring file)
        • ...
      • trusted-x509-keys (optionally holds X.509 certificate files used for signing, see JS7 - Deployment of Scheduling Objects)
        • <x509-certificate> (X.509 certificate file)
        • <x509-certificate> (X.509 certificate file)
        • ...
      • https-keystore.p12 (optional default location and file name of a PKCS12 keystore)
      • https-truststore.p12 (optional default location and file name of a PKCS12 truststore)
      • private.conf (optional configuration file, for example to specify keystore, truststore and Distinguished Names of Controller certificate, see JS7 - Agent Configuration Items)
      • log4j2.xml (optional log configuration file, see JS7 - Log Levels and Debug Options)
    • agent.conf (optional configuration file, see JS7 - Agent Configuration Items)

...

  • <template-name> (arbitrary directory names for templates can be used)
  • <template-name> (arbitrary directory names for templates can be used)
  • ...

...

  • <root-ca-certificate> (the Root CA Certificate file, frequently available with a .pem, .crt extension)

...

  • config (general configuration)
    • private (specific configuration)
      • trusted-pgp-keys (optionally holds PGP public key files and keyring files used for signing, see JS7 - Deployment of Scheduling Objects)
        • <pgp-public-key> (public key file or keyring file)
        • <pgp-public-key> (public key file or keyring file)
        • ...
      • trusted-x509-keys (optionally holds X.509 certificate files used for signing, see JS7 - Deployment of Scheduling Objects)
        • <x509-certificate> (X.509 certificate file)
        • <x509-certificate> (X.509 certificate file)
        • ...
      • https-keystore.p12 (optional default location and file name of a PKCS12 keystore)
      • https-truststore.p12 (optional default location and file name of a PKCS12 truststore)
      • private.conf (optional configuration file, for example to specify keystore, truststore and Distinguished Names of JOC Cockpit certificate, see JS7 - Controller Configuration Items)
      • log4j2.xml (optional log configuration file, see JS7 - Log Levels and Debug Options)
    • controller.conf (optional configuration file, see JS7 - Controller Configuration Items)

...

  • <template-name> (arbitrary directory names for templates can be used)
  • <template-name> (arbitrary directory names for templates can be used)
  • ...

...

  • instances (holds configuration files that are specific for a JOC Cockpit instance)
    • <server>.<instance-type> (holds configuration files for a JOC Cockpit instance running on a specific server with the instance type being primary or secondary)
      • resources (optionally holds configuration files such as the joc.properties file, keystore, truststore files etc.)
      • response (holds response files, mainly the joc_install.xml response file, that are copied to the JOC Cockpit's setup directory)
    • <server>.<instance-type> (holds configuration files for a JOC Cockpit instance running on a specific server with the instance type being primary or secondary)
    • ...
  • templates (holds configuration files that act as templates for a number of JOC Cockpit instances)
    • <template-name> (arbitrary directory names for templates can be used)
      • resources (optionally holds configuration files such as the joc.properties file, keystore, truststore files etc.)
      • response (holds response files, mainly the joc_install.xml response file, that are copied to the JOC Cockpit's setup directory)
    • <template-name> (arbitrary directory names for templates can be used)
    • ...

...

  • ... (users can apply an arbitrary directory hierarchy at this level)
    • js7_agent_unix.<release>.tar.gz (JS7 Agent installation tarball as download from the SOS Web Site)
    • js7_controller_unix.<release>.tar.gz (JS7 Controller installation tarball as download from the SOS Web Site)
    • js7_joc_linux.<release>.tar.gz (JS7 JOC Cockpit installation tarball as download from the SOS Web Site)

...

  • ... (sub-directories used for Agent installation)

...

  • ... (sub-directories used for Controller installation)

...

Explanation on Individual Files

Environment Script: env.sh

The Environment Script initializes a number of environment variables that map to directories as explained above.

...

Code Block
titleExample for Environment File env.sh
linenumberstrue
#!/bin/sh

SCRIPT_HOME=$HOME/js7.deploy

DEP_ARCHIVE="${DEP_ARCHIVE:-$SCRIPT_HOME/archive}"
DEP_BIN="${DEP_BIN:-$SCRIPT_HOME/bin}"
DEP_CONFIG="${DEP_CONFIG:-$SCRIPT_HOME/config}"
DEP_DESC="${DEP_DESC:-$SCRIPT_HOME/desc}"
DEP_INSTALL="${DEP_INSTALL:-$SCRIPT_HOME/install}"
DEP_RELEASE=/mnt/releases/scheduler_setups
DEP_TARGET="/home/sos/js7.target"
DEP_WORK="${DEP_WORK:-$SCRIPT_HOME/work}"

JAVA_HOME="${SCRIPT_HOME}"/jdk-11.0.2
PATH=${JAVA_HOME}/bin:${PATH}

export JAVA_HOME PATH DEP_ARCHIVE DEP_BIN DEP_CONFIG DEP_DESC DEP_INSTALL DEP_RELEASE DEP_TARGET DEP_WORK

Explanation:

  • Any environment variable makes use of a fallback if not otherwise specified.
    • By default the env.sh script's directory is assumed to hold the directory layout of the Deployment Server.
  • Each environment variable in the script points to a directory with a specific purpose that can be located outside of the suggested directory layout.
    • In the above example the DEP_RELEASE environment variable specifies a location of JS7 installation files from a mount point.
    • The DEP_ARCHIVE, DEP_BIN, DEP_CONFIG, DEP_DESC, DEP_WORK environment variables point to directories explained with the directory layout.
    • The DEP_TARGET environment variable points to a location on the same server to simulate deployments to target machines.
  • The JAVA_HOME environment variable is specified to enforce use of a specific Java version for temporary installation of JS7 components during packaging.
  • Consider that environment variables have to be exported in order to be applied by subsequent calls to deployment scripts.

...

Code Block
titleExample for Environment Script env.sh
linenumberstrue
. ./env.sh

...

Code Block
titleExample for Installer Wrapper Script run_install_agent.sh
linenumberstrue
#!/bin/sh

set -e

JS7_DEP_DEPLOY="/home/sos/js7.deploy/config"
JS7_DEP_SCRIPT="/home/sos/js7.deploy/bin"
JS7_DEP_WORK="/home/sos/js7.deploy/work/agents/agent_001"

mkdir -p "/home/sos/js7.deploy/work/agents/agent_001"

${JS7_DEP_SCRIPT}/js7_install_agent.sh \
    --real-path-prefix="${JS7_DEP_WORK}" \
    --release="2.5.0" \
    --tarball="/mnt/releases/scheduler_setups/2.5.0/js7_agent_unix.2.5.0.tar.gz" \
    --home="${JS7_DEP_WORK}/opt/sos-berlin.com/js7/agent-primary" \
    --data="${JS7_DEP_WORK}/var/sos-berlin.com/js7/agent-primary" \
    --http-port="localhost:31443" \
    --https-port="centostest-primary.sos:31443" \
    --controller-id="testsuite" \
    --controller-primary-cert="${JS7_DEP_DEPLOY}/certs/server/centostest-primary.crt" \
    --controller-secondary-cert="${JS7_DEP_DEPLOY}/certs/server/centostest-secondary.crt" \
    --keystore="${JS7_DEP_DEPLOY}/agents/instances/agent_001/config/private/https-keystore.p12" \
    --keystore-password="jobscheduler" \
    --truststore="${JS7_DEP_DEPLOY}/agents/instances/agent_001/config/private/https-truststore.p12" \
    --truststore-password="jobscheduler" \
    --deploy-dir="${JS7_DEP_DEPLOY}/agents/templates/https/config" \
    --make-dirs

Explanation:

  • The parameterization is used from the Deployment Descriptor.
  • The --home and --data directories are prefixed with the directory of the work area.
    • The --real-path-prefix holds the work area directory. The Installer Script makes use of this information to conclude the effective directories as used for the target machine.
  • Wrapper Scripts for Controllers and JOC Cockpit are similarly parameterized.

...

Code Block
titleExample for Deployment Script run_deploy_agent.sh
linenumberstrue
#!/bin/sh

set -e

scp -i "/home/sos/.ssh/sos_rsa" -P 22 "/home/sos/js7.deploy/archive/agent-https-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz" sos@centostest-primary:"/tmp"
ssh -i "/home/sos/.ssh/sos_rsa" -p 22 -t -t -o LogLevel=QUIET sos@centostest-primary << 'EOF'
systemctl stop js7.agent_001
cd "/opt/sos-berlin.com/js7" > /dev/null
test -e "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz" && gzip -c -d < "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz" | tar -xf -
cd - > /dev/null
rc=$?
systemctl start js7.agent_001
exit $rc
EOF

scp -i "/home/sos/.ssh/sos_rsa" -P 22 "/home/sos/js7.deploy/archive/agent-https-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz" sos@centostest-primary:"/tmp"
ssh -i "/home/sos/.ssh/sos_rsa" -p 22 -t -t -o LogLevel=QUIET sos@centostest-primary << 'EOF'
systemctl stop js7.agent_001
cd "/var/sos-berlin.com/js7" > /dev/null
test -e "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz" && gzip -c -d < "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz" | tar -xf -
cd - > /dev/null
rc=$?
systemctl start js7.agent_001
exit $rc
EOF

Explanation:

Changes to the Directory Layout

The directory layout of the Deployment Server is not carved in stone. Instead, users can modify the env.sh Environment Script to point to different directories.

...

Code Block
titleExample for use of an individual script to create deployment packages
linenumberstrue
#!/bin/sh

set -e

SCRIPT_HOME=$(cd "$(dirname "$0")" >/dev/null && pwd)

DEP_ARCHIVE=${DEP_ARCHIVE:-$(dirname "${SCRIPT_HOME}")/archive}
DEP_BIN=${DEP_BIN:-$(dirname "${SCRIPT_HOME}")/bin}
DEP_CONFIG=${DEP_CONFIG:-$(dirname "${SCRIPT_HOME}")/config}
DEP_DESC=${DEP_DESC:-$(dirname "${SCRIPT_HOME}")/desc}
DEP_WORK=${DEP_WORK:-$(dirname "${SCRIPT_HOME}")/work}


${DEP_BIN}/js7_create_deployment_package.sh \
    --deploy-desc=${DEP_DESC}/agent-https-2022-12-04.json \
    --deploy-dir=${DEP_CONFIG} \
    --archive-dir=${DEP_ARCHIVE} \
    --script-dir=${DEP_BIN} \
    --work-dir=${DEP_WORK} \
    --deploy-agent-id=deploymentAgent \
    --keep-script \
    --keep-work

Explanation:

  • The example of an individual script to create deployment packages makes use of existing environment variables to apply individual directories or to fall back to the default directory layout.
  • Use of set -e is recommended in order not to proceed execution of scripts in case of errors.
  • Use of environment variables from the above example allows to execute the script from any working directory.

Summary of Steps to perform Deployment

Deployment is an operation that is performed in the following steps:

...

...

      • the <archive>/<deployment-descriptor>

...

      • directory to JOC Cockpit, for details see JS7 -

...

      • Deployment Workflow.
      • Check the imported workflow and adjust to your needs, for example assignment of an Agent with access to the deployment directories in the Deployment

...

      • Area.
      • Adjust the imported schedule to your needs should you want to schedule execution of deployments

...

      • for a specific date and time. Alternatively you can add an ad hoc order to the Deployment Workflow from the JOC Cockpit's Workflows view. This step offers to copy the parameterization available from the schedule to the current

...

      • order

...

      • .

Resources

...