JS7 JobScheduler

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

Introduction

For JS7 - Automated Deployment use of a Deployment Area is recommended.

The JS7 - Deployment Area is used for the following purposes:

  • to hold the script environment for JS7 - Automated Installation and Update and for JS7 - Deployment Packaging,
  • to hold the configuration files and optionally certificates for deployment of JS7 components,
  • to hold the JS7 installations per JS7 release and component such as JOC Cockpit, Controller, Agent,
  • to hold the archive of Deployment Packages per JS7 releases and target host,

Preparing the Deployment Area

Setting up the Deployment Area

Download of Deployment Area Tarball

A tarball for the directories and scripts of the Deployment Area is provided:

Download tarball: 

Download Deployment Area tarball from SOS Web Site
curl "https://download.sos-berlin.com/JobScheduler/js7_deploy.tar.gz" --output "js7_deploy.tar.gz"

Extracting the Deployment Area Tarball

Extract the Deployment Area tarball to an arbitrary directory. The tarball includes the top-level directory js7.deploy, therefore the following command will extract to $HOME/js7.deploy.

Extract Deployment Area tarball
cd $HOME
tar -xzf js7.deploy.tar.gz


After extraction users find the following directory layout:

  • js7.deploy
    • archive (empty, holds the resulting Deployment Packages)
    • bin (holds Scripts)
    • ca (holds the Certificate Authority)
    • config (holds configuration templates)
    • desc (holds JS7 - Deployment Descriptor Examples)
    • release (empty, used for JS7 installation tarballs)
    • work (empty, used for temporary installation)
    • env.sh (Environment Script)

Find more details about included directories and files from the JS7 - Deployment Area - Directory Layout article.

Users can rename and relocate the top-level folder js7.deploy at their will.

Setting up the Environment Script

The env.sh Environment Script is extracted to the js7.deploy top-level folder. The script creates a number of environment variables that indicate the directories in use.

Adjusting Locations in the Environment Script

Users can adjust the location of directories to their needs:

Environment Script env.sh
#!/bin/sh

SCRIPT_HOME=${HOME}/js7.deploy

DEP_ARCHIVE="${DEP_ARCHIVE:-$SCRIPT_HOME/archive}"
DEP_BIN="${DEP_BIN:-$SCRIPT_HOME/bin}"
DEP_CA="${DEP_CA:-$SCRIPT_HOME/ca}"
DEP_CONFIG="${DEP_CONFIG:-$SCRIPT_HOME/config}"
DEP_DESC="${DEP_DESC:-$SCRIPT_HOME/desc}"
DEP_RELEASE="${DEP_RELEASE:-$SCRIPT_HOME/release}"
DEP_WORK="${DEP_WORK:-$SCRIPT_HOME/work}"

JAVA_HOME="/usr/lib/jvm/jdk-11.0.2"
PATH=${JAVA_HOME}/bin:${PATH}

export JAVA_HOME PATH DEP_ARCHIVE DEP_BIN DEP_CA DEP_CONFIG DEP_DESC DEP_RELEASE DEP_WORK


Explanation:

  • Line 3: The top-level directory of the Deployment Area is specified. By default the js7.deploy directory in the user's home directory is assumed.
  • Line 5-11: The environment variables indicate the location of the directories extracted from the Deployment Area tarball. Users who wish a different location adjust the variables to their needs.
    • The default values act as a fallback if one of the DEP_* environment variables is not set.
    • Users can specify an absolute path, for example: DEP_RELEASE=/mnt/releases/js7 if the location of JS7 installation tarballs should be different from the default location.
  • Line 13-14: The location of the Java JDK is specified with the JAVA_HOME environment variable. Users have to adjusts the environment variable to the location of their Java JDK.
    • Typically Java LTS releases 11 and newer can be used. For Agent and Controller Java 1.8 can be used. For details see Which Java versions is JobScheduler available for?
    • The PATH environment variable is adjusted to point to the location of the bin sub-directory of Java.
  • Line 16: The environment variables are exported and made available for use with the Deployment Area scripts.

Running the Environment Script

The Environment Script has to be executed to make environment variables know to the Deployment Area scripts.

The script has to be sourced, this means the call to the script is preceded by a dot and a space like this:

Run the env.sh Environment Script
. ./env.sh

Download JS7 Release

It is good practice to download the installation .tar.gz files for the desired JS7 releases from the SOS Web Site and to store them to a directory hierarchy like this:

  • js7.deploy
    • release
      • 2.4.0
        • js7_agent_unix.2.4.0.tar.gz
        • js7_controller_unix.2.4.0.tar.gz
        • js7_joc_linux.2.4.0.tar.gz
      • 2.5.0
        • js7_agent_unix.2.5.0.tar.gz
        • js7_controller_unix.2.5.0.tar.gz
        • js7_joc_linux.2.5.0.tar.gz

Users are free to choose the directory hierarchy at their will.

Download JS7 Installation Archives

Download can be performed in a number of ways. The curl utility can be used for this purpose.

Download JS7 Release from SOS Web Site
curl "https://download.sos-berlin.com/JobScheduler.2.5/js7_agent_unix.2.5.0.tar.gz" --output "js7_agent_unix.2.5.0.tar.gz"
curl "https://download.sos-berlin.com/JobScheduler.2.5/js7_controller_unix.2.5.0.tar.gz" --output "js7_controller_unix.2.5.0.tar.gz"
curl "https://download.sos-berlin.com/JobScheduler.2.5/js7_joc_linux.2.5.0.tar.gz" --output "js7_joc_linux.2.5.0.tar.gz"

Managing Deployment Packages

Management of Deployment Packages includes the following steps:

  • Deployment Packages are specified from a JS7 - Deployment Descriptor that holds the information which JS7 components should be installed, on which hosts to install the components and what configuration items should be used.
  • Deployment Packages are created from a script available with JS7 - Deployment Packaging.
  • Deployment Packages are transferred and extracted to target hosts. 

Creating Deployment Descriptors

The Deployment Descriptor is specified from a .json file.

Users find examples in the desc sub-directory of the Deployment Area:

Deployment Descriptors for use with Controller and JOC Cockpit Standalone Instances

Deployment Descriptors in this section are used with Standalone instances of Controller and JOC Cockpit.

The following Deployment Descriptors ship with the Deployment Area:

  • js7.deploy
    • desc
      • standalone
        • standalone-agent-http-2022-12-04.json
        • standalone-agent-https-2022-12-04.json
        • standalone-controller-http-2022-12-04.json
        • standalone-controller-https-2022-12-04.json
        • standalone-joc-http-2022-12-04.json
        • standalone-joc-https-2022-12-04.json
        • standalone-joc-controller-agent-https-2022-12-04.json

Deployment Descriptors for use with Controller and JOC Cockpit Cluster Instances

Deployment Descriptors in this section are used with Cluster instances of Controller and JOC Cockpit that are subject to the JS7 - License. 

The following Deployment Descriptors ship with the Deployment Area:

  • js7.deploy
    • desc
      • cluster
        • cluster-agent-http-2022-12-04.json
        • cluster-agent-https-2022-12-04.json
        • cluster-controller-http-2022-12-04.json
        • cluster-controller-https-2022-12-04.json
        • cluster-joc-http-2022-12-04.json
        • cluster-joc-https-2022-12-04.json
        • cluster-joc-controller-agent-https-2022-12-04.json

Adjusting the Deployment Descriptor

For starters it is recommended to use the standalone-agent-http-2022-12-04.json Deployment Descriptor that looks similar to this:

Deployment Descriptor standalone-agent-http-2022-12-04.json
{
    "descriptor": {
        "descriptorId": "standalone-agent-http-2022-12-04",
        "title": "Install or update Agent Standalone instance using HTTP connections",
        "account": "sos",
        "scheduled": "2022-12-02T06:19:12+01:00",
        "created": "2022-11-30T23:19:12+01:00"
    },
    "agents": [
        {
            "agent_001": {
                "target": {
                    "connection": {
                        "host": "centostest-primary",
                        "port": 22
                    },
                    "authentication": {
                        "method": "publickey",
                        "user": "sos",
                        "keyFile": "/home/sos/.ssh/sos_rsa"
                    },
                    "packageLocation": "/tmp"
                },
                "media": {
                    "release": "2.5.0",
                    "tarball": "${DEP_RELEASE}/2.5.0/js7_agent_unix.2.5.0.tar.gz"
                },
                "installation": {
                    "home": "/opt/sos-berlin.com/js7/agent-standalone",
                    "data": "/var/sos-berlin.com/js7/agent-standalone",
                    "httpPort": "31445",
                    "javaHome": "/opt/java/jdk8u202-b08"
                }
            }
        }
    ],
    "controllers": [],
    "joc": []
}


Explanation:

  • The descriptor.descriptorId element identifies the descriptor. The identifier is similarly used as the name of the Deployment Package. Identifiers can be made up from lowercase and uppercase letters, can include digits, underscores and hyphens. The file name of the Deployment Descriptor is not relevant.
  • The agents.agent_001 element specifies the unique Agent name agent_001.
    • Agent names can be made up from lowercase and uppercase letters, can include digits, underscores and hyphens. Users are free to choose Agent names, for example from the hostname or FQDN that the Agent is operated for, for example using element names such as agents.centostest-primary, agents.centostest-secondary.sos etc.
    • Any number of Agents can be specified in a Deployment Descriptor. 
  • The target section specifies the connection and authentication details for the target host to which the Agent should be deployed.
    • Deployment is performed using scp and ssh commands that will use the indicated SSH host and port.
    • Authentication makes use of public/private key specifying the user account and the location of the private keyFile. This setting can be omitted if the private key file matches one of the known names such as id_rsa.
    • For automated deployment It is required that the deployment user account can connect to the target host without use of a password but with a private key only.
    • The packageLocation element specifies the location where the Deployment Package tarball will be stored.
  • The media section specifies the release number and the location of the JS7 installation tarball for the Agent. The DEP_RELEASE environment variable from the env.sh script can be used to specify the directory in which the installation tarball is available.
  • The installation section specifies installation options for the Agent:
    • The Agent's home directory has to be specified. A data directory can optionally be specified, if it is omitted then <home>var_<http-port> is assumed. Consider that the target.user account has permissions to create the directories.
    • By default the directories and files of the extracted Deployment Package are owned by the deployment account specified with the target.user element. Should a different account be used on the target host then
      • the owner account of the home directory can be specified with the installation.homeOwner element.
      • the owner account of the data directory can be specified with the installation.dataOwner element.
      • the run-time account of the Agent's service/daemon can be specified with the installation.runUser element.
    • The Agent's HTTP port can be specified that defaults to port 4445.
    • The location of the Java JDK on the target host can be specified with the javaHome element.
      • The location of Java will be added to the Agent's Start Script in <home>/bin/agent_<http-port>.sh and to the systemd service file in <home>/bin/agent_<http-port>.service.
      • If this setting is omitted, then the Java location is determined by the run-time account, for example from its OS profile settings.

Creating the Deployment Package

With the Deployment Descriptor being available the Deployment Package can be created from a script. The Deployment Package will be stored to the js7.deploy/archive directory.

The Deployment Packaging script performs the following steps:

  • install the JS7 components according to the Deployment Descriptor
  • create a deployment tarball of the installed files. If different home and data directories are used then a tarball for each directory will be created.
  • create a Deployment Script that will transfer and extract the deployment tarball on the target host.

Running the Deployment Packaging Script

The Deployment Packaging Script is available from the js7.deploy/bin directory.

The script is parameterized for example like this:

  • --deploy-desc: specifies the location of the Deployment Descriptor,
  • --make-dirs: specifies that sub-directories should be created if they do not exist.

Find details about parameterization from the JS7 - Deployment Packaging article. 

Run the Deployment Packaging Script
${DEP_BIN}/js7_create_deployment_package.sh \
    --deploy-desc=${DEP_DESC}/standalone/standalone-agent-http-2022-12-04 \
    --make-dirs


Output of the script will look like this:

Run the Deployment Packaging Script
-- begin of log --------------
/home/sos/js7.deploy/bin/js7_create_deployment_package.sh --deploy-desc=/home/sos/js7.deploy/desc/standalone/standalone-agent-http-2022-12-04.json --make-dirs
-- begin of output -----------
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
.. creating deployment package for Agent ID: agent_001
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
.... creating install script: /home/sos/js7.deploy/work/tmp/js7_deploy_package_install_17858.sh
.... running install script: /home/sos/js7.deploy/work/tmp/js7_deploy_package_install_17858.sh
-- begin of log --------------
/home/sos/js7.deploy/bin/js7_install_agent.sh --real-path-prefix=/home/sos/js7.deploy/work/agents/agent_001 --release=2.5.0 --tarball=/home/sos/js7.deploy/release/2.5.0/js7_agent_unix.2.5.0.tar.gz --home=/home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone --data=/home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone --http-port=31445 --java-home=/opt/java/jdk8u202-b08 --make-dirs
-- begin of output -----------
.. using tarball: /home/sos/js7.deploy/release/2.5.0/js7_agent_unix.2.5.0.tar.gz
.. extracting tarball to temporary directory: /tmp/js7_install_agent_18024.tmp
.. creating Agent home directory: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone
.. creating Agent data directory: /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone
.. copying files from extracted tarball directory: /tmp/js7_install_agent_18024.tmp/agent to Agent home: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone
.. creating Agent data directory: /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone
.. copying files to Agent data directory: /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone
.. copying Agent Instancee Start Script /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_instance.sh-example to /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.sh
.. copying /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent.service-example to /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.service
.. updating Agent Instance Start Script: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.sh
.. updating Agent systemd service file: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.service
-- end of log ----------------
.... creating installation deployment package file: tar -cf /home/sos/js7.deploy/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.install.tar -C /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7 agent-standalone
.... installation deployment package ready: /home/sos/js7.deploy/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz
.... creating configuration deployment package file: tar -cf /home/sos/js7.deploy/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.config.tar -C /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7 agent-standalone
.... configuration deployment package ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz
.... install script for deployment package ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/run_install_agent.sh
.... job script for deployment package ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/run_deploy_agent.sh
.... creating JS7 import file: tar -cf /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/js7_import.tar *
.... JS7 import file ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/js7_import.tar.gz
.. Deployment Script ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/run_deploy.sh
-- end of log ----------------


As a result the Deployment Package is available from the following directory hierarchy:

  • js7.deploy
    • archive
      • standalone-agent-http-2022-12-04 (directory name created from the Deployment Descriptor ID)
        • agents (sub-directory for Agent deployments)
          • agent_001 (sub-directory for specific Agent identified by its Agent ID)
            • js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz (tarball of the Agent's configuration directory)
            • js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz (tarball of the Agent's installation directory)
            • run_deploy_agent.sh (Deployment Script for Agent)
            • run_install_agent.sh (Installer Wrapper Script for Agent)
        • js7_import.tar.gz (tarball for import into JS7 JOC Cockpit, includes workflow and schedule for automated deployment)
        • run_deploy.sh (Deployment Script, includes calls to agents/agent_001/run_deploy_agent.sh and similar scripts per JS7 component included) 

 Basically the run_deploy.sh script can be used to transfer the Agent's tarballs for configuration and installation directories to the target host.

  • Use of the Deployment Script from the command line is applicable if a smaller number of JS7 components should be installed and if no JS7 scheduling environment is yet available.
  • If a JS7 scheduling environment is set up then it is preferable to import the js7_import.tar.gz to JOC Cockpit that adds a workflow and schedule for parallel deployment of JS7 components. For details see JS7 - Deployment Workflow.

Performing Deployment

Deployment can be performed by executing the Deployment Script from the command line. 

Run the Deployment Script
${DEP_ARCHIVE}/standalone-agent-http-2022-12-04/run_deploy.sh




  • No labels