JS7 JobScheduler

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Introduction

For JS7 - Automated Deployment use of a Deployment Area is recommended.

The JS7 - Deployment Area is used for the following purposes:

  • to hold the script environment for JS7 - Automated Installation and Update and for JS7 - Deployment Packaging,
  • to hold the configuration files and optionally certificates for deployment of JS7 components,
  • to hold the JS7 installations per JS7 release and component such as JOC Cockpit, Controller, Agent,
  • to hold the archive of Deployment Packages per JS7 releases and target host,

As a prerequisite to managing Deployment Packages the Deployment Area has to be in place, see JS7 - Deployment - Setting up the Deployment Area.

Managing Deployment Packages

Management of Deployment Packages includes the following steps:

  • Deployment Packages are specified from a JS7 - Deployment Descriptor that holds the information which JS7 components should be installed, on which hosts to install the components and what configuration items should be used.
  • Deployment Packages are created from a script available with JS7 - Deployment Packaging.
  • Deployment Packages are transferred and extracted to target hosts. 

Creating Deployment Descriptors

The Deployment Descriptor is specified from a .json file.

Users find examples in the desc sub-directory of the Deployment Area:

Deployment Descriptors for use with Controller and JOC Cockpit Standalone Instances

Deployment Descriptors in this section are used with Standalone instances of Controller and JOC Cockpit.

  • The http file name part indicates that HTTP connections are used.
  • The https file name part indicates that HTTPS connections and certificates are used. Such descriptors require prior creation of certificates, see JS7 - Deployment - Managing Certificates.

The following Deployment Descriptors ship with the Deployment Area:

  • js7.deploy
    • desc
      • standalone
        • standalone-agent-http-2022-12-04.json
        • standalone-agent-https-2022-12-04.json
        • standalone-controller-http-2022-12-04.json
        • standalone-controller-https-2022-12-04.json
        • standalone-joc-http-2022-12-04.json
        • standalone-joc-https-2022-12-04.json
        • standalone-joc-controller-agent-https-2022-12-04.json

Deployment Descriptors for use with Controller and JOC Cockpit Cluster Instances

Deployment Descriptors in this section are used with Cluster instances of Controller and JOC Cockpit that are subject to the JS7 - License. 

  • The http file name part indicates that HTTP connections are used.
  • The https file name part indicates that HTTPS connections and certificates are used. Such descriptors require prior creation of certificates, see JS7 - Deployment - Managing Certificates.

The following Deployment Descriptors ship with the Deployment Area:

  • js7.deploy
    • desc
      • cluster
        • cluster-agent-http-2022-12-04.json
        • cluster-agent-https-2022-12-04.json
        • cluster-controller-http-2022-12-04.json
        • cluster-controller-https-2022-12-04.json
        • cluster-joc-http-2022-12-04.json
        • cluster-joc-https-2022-12-04.json
        • cluster-joc-controller-agent-https-2022-12-04.json

Adjusting the Deployment Descriptor

For starters it is recommended to use the standalone-agent-http-2022-12-04.json Deployment Descriptor that looks similar to this:

Deployment Descriptor standalone-agent-http-2022-12-04.json
{
    "descriptor": {
        "descriptorId": "standalone-agent-http-2022-12-04",
        "title": "Install or update Agent Standalone instance using HTTP connections",
        "account": "sos",
        "scheduled": "2022-12-02T06:19:12+01:00",
        "created": "2022-11-30T23:19:12+01:00"
    },
    "agents": [
        {
            "agent_001": {
                "target": {
                    "connection": {
                        "host": "centostest-primary",
                        "port": 22
                    },
                    "authentication": {
                        "method": "publickey",
                        "user": "sos",
                        "keyFile": "/home/sos/.ssh/sos_rsa"
                    },
                    "packageLocation": "/tmp"
                },
                "media": {
                    "release": "2.5.0",
                    "tarball": "2.5.0/js7_agent_unix.2.5.0.tar.gz"
                },
                "installation": {
                    "home": "/opt/sos-berlin.com/js7/agent-standalone",
                    "data": "/var/sos-berlin.com/js7/agent-standalone",
                    "httpPort": "31445",
                    "javaHome": "/opt/java/jdk8u202-b08"
                }
            }
        }
    ],
    "controllers": [],
    "joc": []
}


Explanation:

  • The descriptor.descriptorId element identifies the descriptor. The identifier is similarly used as the name of the Deployment Package. Identifiers can be made up from lowercase and uppercase letters, can include digits, underscores and hyphens. The file name of the Deployment Descriptor is not relevant.
  • The agents.agent_001 element specifies the unique Agent name agent_001.
    • Agent names can be made up from lowercase and uppercase letters, can include digits, underscores and hyphens. Users are free to choose Agent names, for example from the hostname or FQDN that the Agent is operated for, for example using element names such as agents.centostest-primary, agents.centostest-secondary.sos etc.
    • Any number of Agents can be specified in a Deployment Descriptor. 
  • The target section specifies the connection and authentication details for the target host to which the Agent should be deployed.
    • Deployment is performed using scp and ssh commands that will use the indicated SSH host and port.
    • Authentication makes use of public/private key specifying the user account and the location of the private keyFile. This setting can be omitted if the private key file matches one of the known names such as id_rsa.
    • For automated deployment It is required that the deployment user account can connect to the target host without use of a password but with a private key only.
    • The packageLocation element specifies the location where the Deployment Package tarball will be stored.
  • The media section specifies the release number and the location of the JS7 installation tarball for the Agent. If a relative path is specified then the value of the DEP_RELEASE environment variable) from the env.sh Environment Script) is prepended to specify the directory in which the installation tarball is available.
  • The installation section specifies installation options for the Agent:
    • The Agent's home directory has to be specified. A data directory optionally can be specified, if it is omitted then the <home>/var_<http-port> directory is assumed. Users should take care that the target.user account has permissions to create required directories.
    • By default the directories and files of the extracted Deployment Package are owned by the deployment account specified with the target.user element.
      • Should a different account be used on the target host then
        • the owner account of the home directory can be specified with the installation.homeOwner element.
        • the owner account of the data directory can be specified with the installation.dataOwner element.
        • the run-time account of the Agent's service/daemon can be specified with the installation.runUser element.
      • In order to modify accounts of directory owners the target.user account has to be provided permissions to perform sudo to switch to the root without using a password.
    • The Agent's HTTP port can be specified that defaults to port 4445.
    • The location of the Java JDK on the target host can be specified with the javaHome element.
      • The location of Java will be added to the Agent's Start Script in <home>/bin/agent_<http-port>.sh and to the systemd service file in <home>/bin/agent_<http-port>.service.
      • If this setting is omitted, then the Java location is determined by the run-time account, for example from its OS profile settings.

Creating the Deployment Package

With the Deployment Descriptor being available the Deployment Package can be created from a script. The Deployment Package will be stored to the js7.deploy/archive directory.

The Deployment Packaging script performs the following steps:

  • install the JS7 components according to the Deployment Descriptor
  • create a deployment tarball of the installed files. If different home and data directories are used then a tarball for each directory will be created.
  • create a Deployment Script that will transfer and extract the deployment tarball on the target host.

Running the Deployment Packaging Script

The Deployment Packaging Script is available from the js7.deploy/bin directory.

The script is parameterized for example like this:

  • --deploy-desc: specifies the location of the Deployment Descriptor,
  • --make-dirs: specifies that sub-directories should be created if they do not exist.

Find details about parameterization from the JS7 - Deployment Packaging article. 

Running the Deployment Packaging Script
${DEP_BIN}/js7_create_deployment_package.sh \
    --deploy-desc=${DEP_DESC}/standalone/standalone-agent-http-2022-12-04 \
    --make-dirs


Output of the script will look like this:

Output of Deployment Packaging Script Run
-- begin of log --------------
/home/sos/js7.deploy/bin/js7_create_deployment_package.sh --deploy-desc=/home/sos/js7.deploy/desc/standalone/standalone-agent-http-2022-12-04.json --make-dirs
-- begin of output -----------
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
.. creating deployment package for Agent ID: agent_001
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
.... creating install script: /home/sos/js7.deploy/work/tmp/js7_deploy_package_install_17858.sh
.... running install script: /home/sos/js7.deploy/work/tmp/js7_deploy_package_install_17858.sh
-- begin of log --------------
/home/sos/js7.deploy/bin/js7_install_agent.sh --real-path-prefix=/home/sos/js7.deploy/work/agents/agent_001 --release=2.5.0 --tarball=/home/sos/js7.deploy/release/2.5.0/js7_agent_unix.2.5.0.tar.gz --home=/home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone --data=/home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone --http-port=31445 --java-home=/opt/java/jdk8u202-b08 --make-dirs
-- begin of output -----------
.. using tarball: /home/sos/js7.deploy/release/2.5.0/js7_agent_unix.2.5.0.tar.gz
.. extracting tarball to temporary directory: /tmp/js7_install_agent_18024.tmp
.. creating Agent home directory: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone
.. creating Agent data directory: /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone
.. copying files from extracted tarball directory: /tmp/js7_install_agent_18024.tmp/agent to Agent home: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone
.. creating Agent data directory: /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone
.. copying files to Agent data directory: /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7/agent-standalone
.. copying Agent Instancee Start Script /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_instance.sh-example to /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.sh
.. copying /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent.service-example to /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.service
.. updating Agent Instance Start Script: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.sh
.. updating Agent systemd service file: /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.service
-- end of log ----------------
.... creating installation deployment package file: tar -cf /home/sos/js7.deploy/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.install.tar -C /home/sos/js7.deploy/work/agents/agent_001/opt/sos-berlin.com/js7 agent-standalone
.... installation deployment package ready: /home/sos/js7.deploy/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz
.... creating configuration deployment package file: tar -cf /home/sos/js7.deploy/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.config.tar -C /home/sos/js7.deploy/work/agents/agent_001/var/sos-berlin.com/js7 agent-standalone
.... configuration deployment package ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz
.... install script for deployment package ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/run_install_agent.sh
.... job script for deployment package ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/run_deploy_agent.sh
.... creating JS7 import file: tar -cf /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/js7_import.tar *
.... JS7 import file ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/js7_import.tar.gz
.. Deployment Script ready: /home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/run_deploy.sh
-- end of log ----------------


As a result the Deployment Package is available from the following directory hierarchy:

  • js7.deploy
    • archive
      • standalone-agent-http-2022-12-04 (directory name created from the Deployment Descriptor ID)
        • agents (sub-directory for Agent deployments)
          • agent_001 (sub-directory for specific Agent identified by its Agent ID)
            • js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz (tarball of the Agent's configuration directory)
            • js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz (tarball of the Agent's installation directory)
            • run_deploy_agent.sh (Deployment Script for Agent)
            • run_install_agent.sh (Installer Wrapper Script for Agent)
            • run_rollout_agent.sh (Rollout Script for Agent)
        • js7_import.tar.gz (tarball for import into JS7 JOC Cockpit, includes workflow and schedule for automated deployment)
        • run_rollout.sh (Rollout Script for any JS7 components, includes calls to agents/agent_001/run_rollout_agent.sh and similar scripts per JS7 component) 

Basically the run_rollout_agent.sh script can be used to transfer the Agent's tarballs for configuration and installation directories to the target host. In addition, the run_deploy_agent.sh script is transferred to the target host and is executed on that host to extract the tarballs of Agent installation and configuration directories.

  • Use of the Rollout Script from the command line is applicable if a smaller number of JS7 components should be installed and if no JS7 scheduling environment is yet available.
  • If a JS7 scheduling environment is set up then it is preferable to import the js7_import.tar.gz tarball to JOC Cockpit that adds a workflow and schedule for parallel deployment of JS7 components. For details see JS7 - Deployment Workflow.

Performing Deployment

Deployment can be performed by

  • importing the workflow and schedule created with the Deployment Package into JOC Cockpit und using the JS7 - Deployment Workflow.
  • or by executing the Rollouot Script from the command line.

The Rollout Script is executed from the js7.deploy/archive directory without further arguments. The script is located in a sub-directory specified by the Deployment Descriptor:

Running the Rollout Script
${DEP_ARCHIVE}/standalone-agent-http-2022-12-04/run_rollout.sh


The script will perform the following actions:

  • When connecting to the target host the script will stop the Agent's systemd service provided that the Deployment Descriptor specifies the target.execPre element with a value StopService.
  • For each tarball that holds the Agent's installation directory and configuration directory the script will
    • transfer the tarball to the target host and location indicated with the Deployment Descriptor (target.packageLocation),
    • extract the tarball to the respective installation or configuration directory as indicated with the Deployment Descriptor (installation.home, installation.data),
    • change ownership of the extracted directories provided that the Deployment Descriptor specifies the installation.homeOwner and installation.dataOwner elements.
  • After transfer and extraction of tarballs the script will start the Agent's systemd service provided that the Deployment Descriptor specifies the target.execPost element with a value StartService.

Resources


  • No labels