JS7 JobScheduler

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Introduction

For JS7 - Automated Deployment use of a Deployment Area is recommended.

The Deployment Area is used for the following purposes:

The following sections explain the directories that are included in a Deployment Area.

Directory Layout

Directories and Files

The suggested directory layout for the Deployment Area includes the following directories and files:

  • archive (holds Deployment Packages)
    • <deployment-descriptor> (specifies the Deployment Descriptor)
      • agents (holds Agent Deployment Packages)
        • <agent-id> (specifies the Agent ID)
          • js7_deploy_agent_unix.<agent-id>.<release>.config.tar.gz (tarball for the Agent's configuration directory)
          • js7_deploy_agent_unix.<agent-id>.<release>.install.tar.gz (tarball for the Agent's installation directory)
          • run_deploy_agent.sh (Deployment Script)
          • run_install_agent.sh (wrapper script for the parameterized call to the js7_install_agent.sh Installer Script)
        • <agent-id> (specifies the Agent ID)
        • ...
      • controllers (holds Controller Deployment Packages)
        • <controller-id> (specifies the Controller ID)
          • <instance-type> (specifies the type of the Controller instance which is primary or secondary) 
            • js7_deploy_controller_unix.<controller-id>.<instance-type>.<release>.config.tar.gz (tarball for the Controller instance's configuration directory)
            • js7_deploy_controller_unix.<controller-id>.<instance-type>.<release>.install.tar.gz (tarball for the Controller instance's installation directory)
            • run_deploy_controller.sh (Deployment Script)
            • run_install_controller.sh (Wrapper Script for the parameterized call to the js7_install_controller.sh Installer Script)
          • <instance-type> (specifies the type of the Controller instance which is primary or secondary) 
        • <controller-id> (specifies the Controller ID)
        • ...
      • joc (holds JOC Cockpit Deployment Packages)
        • <joc-id> (specifies the JOC Cockpit ID)
          • <instance-type> (specifies the type of the JOC Cockpit instance which is primary or secondary) 
            • js7_deploy_joc_linux.<joc-id>.<instance-type>.<release>.config.tar.gz (tarball for the JOC Cockpit instance's configuration directory)
            • js7_deploy_joc_linux.<joc-id>.<instance-type>.<release>.install.tar.gz (tarball for the JOC Cockpit instance's installation directory)
            • run_deploy_joc.sh (Deployment Script)
            • run_install_joc.sh (Wrapper Script for the parameterized call to the js7_install_joc.sh Installer Script.
          • <instance-type> (specifies the type of the JOC Cockpit instance which is primary or secondary) 
        • <joc-id> (specifies the JOC Cockpit ID)
        • ...
      • js7_import_tar.gz (holds the Deployment Workflow for import to JS7 JOC Cockpit)
      • run_deploy.sh (Wrapper Script to run all run_deploy_*.sh Deployment Scripts for Agent, Controller and JOC Cockpit instances)
    • <deployment-descriptor> (specifies the Deployment Descriptor)
    • ...
  • bin (holds executable files, preferably individual scripts, Deployment Scripts and Installer Scripts available from JS7 - Download)
  • ca  (holds the Certificate Authority as explained from  JS7 - How to create self-signed Certificates, not used if an external Certificate Authority is in place)
    • certs (holds CA-signed Certificates)
    • csr (holds Certificate Signing Requests)
    • private (holds Private Keys)
  • config (holds configuration files)
    • agents (holds Agent configuration files)
      • instances (holds configuration files specific for an Agent)
        • <agent-id> (specifies the Agent ID for directories and files that are specific to an Agent)
          • config (general configuration)
            • private (specific configuration)
              • https-keystore.p12 (default location and file name of a PKCS12 keystore holding the Agent's private key and certificate)
              • https-truststore.p12 (default location and file name of a PKCS12 truststore holding the Root CA Certificate)
        • <agent-id> (specifies the Agent ID for directories and files that are specific to an Agent)
        • ...
      • templates (holds configuration files that act as templates for a number of Agents)
        • <template-name> (an arbitrary directory name for templates can be used)
        • <template-name> (an arbitrary directory name for templates can be used)
        • ...
    • controllers (holds Controller configuration files)
      • instances (holds configuration files specific for Controller instances)
        • <controller-id>.<controller-type> (specifies the Controller ID for directories and files that are specific to a Controller instance with the instance type being primary or secondary)
          • config (general configuration)
            • private (specific configuration)
              • <controller-cert> (the Controller instance's server authentication certificate)
              • https-keystore.p12 (default location and file name of a PKCS12 keystore holding the Controller instance's private key and certificate)
              • https-truststore.p12 (default location and file name of a PKCS12 truststore holding the Root CA Certificate)
        • <controller-id>.<controller-type> (specifies the Controller ID for directories and files that are specific to a Controller)
        • ...
      • templates (holds configuration files that act as templates for a number of Controllers)
        • <template-name> (an arbitrary directory name for templates can be used)
        • <template-name> (an arbitrary directory name for templates can be used)
        • ...
    • joc  (holds JOC Cockpit configuration files)
      • instances (holds configuration files that are specific for a JOC Cockpit instance)
        • <joc-id>.<instance-type> (holds configuration files for a JOC Cockpit instance with the instance type being primary or secondary)
          • resources (optionally holds configuration files such as the joc.properties file, keystore, truststore files etc.)
          • response (optionally holds response files, mainly the joc_install.xml response file, that are copied to the JOC Cockpit's setup directory)
        • <joc-id>.<instance-type> (holds configuration files for a JOC Cockpit instance with the instance type being primary or secondary)
        • ...
      • templates (holds configuration files that act as templates for a number of JOC Cockpit instances)
        • <template-name> (an arbitrary directory name for templates can be used)
          • resources (optionally holds configuration files such as the joc.properties file)
          • response (holds response files, mainly the joc_install.xml response file, that are copied to the JOC Cockpit's setup directory)
        • <template-name> (an arbitrary directory name for templates can be used)
        • ...
    • licenses (optionally holds files related to JS7 licenses)
  • desc (holds Deployment Descriptors)
    • <deployment-descriptor>.json (Deployment Descriptor .json file)
    • <deployment-descriptor>.json (Deployment Descriptor .json file)
    • ...
  • logs (holds log files)
    • deployment_package.<deployment-descriptor>.<host>.<timestamp>.log (Packaging Script log files)
    • install_js7_agent.<host>.<timestamp>.log (Agent Installer log files)
    • install_js7_controller.<host>.<timestamp>.log (Controller Installer log files)
    • install_js7_joc.<host>.<timestamp>.log (JOC Cockpit Installer log files)
  • release  (holds the installation tarballs of JS7 releases)
    • ... (users can apply an arbitrary directory hierarchy at this level)
      • js7_agent_unix.<release>.tar.gz (JS7 Agent installation tarball as download from the SOS Web Site)
      • js7_controller_unix.<release>.tar.gz (JS7 Controller installation tarball as download from the SOS Web Site)
      • js7_joc_linux.<release>.tar.gz (JS7 JOC Cockpit installation tarball as download from the SOS Web Site)
  • work (the working area is preferably used to perform installation of JS7 components during packaging)
    • agents (directory for Agent installation during packaging)
      • <agent-id> (specifies the Agent ID for directories and files that are specific to an Agent)
        • ... (sub-directories used for Agent installation)
      • <agent-id> (specifies the Agent ID for directories and files that are specific to an Agent)
      • ...
    • controllers (directory for Controller installation during packaging)
      • <controller-id> (specifies the Controller ID for directories and files that are specific to a Controller)
        • ... (sub-directories used for Controller installation)
      • <controller-id> (specifies the Controller ID for directories and files that are specific to a Controller)
      • ...
    • tmp (temporary files are written to this directory, if the --keep-work switch is used when invoking the JS7 - Deployment Packaging then files will remain in this directory which suggests cleanup by the user)
  • env.sh (Environment Script, see next chapter)

Explanation on Individual Files

Environment Script: env.sh

The Environment Script initializes a number of environment variables that map to directories as explained above.

The script has to be invoked before running any script to install, to package or to deploy packages to target hosts.

Example for Environment File env.sh
#!/bin/sh

SCRIPT_HOME=${HOME}/js7.deploy

DEP_ARCHIVE="${DEP_ARCHIVE:-$SCRIPT_HOME/archive}"
DEP_BIN="${DEP_BIN:-$SCRIPT_HOME/bin}"
DEP_CA="${DEP_CA:-$SCRIPT_HOME/ca}"
DEP_CONFIG="${DEP_CONFIG:-$SCRIPT_HOME/config}"
DEP_DESC="${DEP_DESC:-$SCRIPT_HOME/desc}"
DEP_RELEASE="${DEP_RELEASE:-$SCRIPT_HOME/release}"
DEP_WORK="${DEP_WORK:-$SCRIPT_HOME/work}"

JAVA_HOME="/usr/lib/jvm/jdk-11.0.2"
PATH=${JAVA_HOME}/bin:${PATH}

export JAVA_HOME PATH DEP_ARCHIVE DEP_BIN DEP_CA DEP_CONFIG DEP_DESC DEP_RELEASE DEP_WORK


Explanation:

  • Any environment variable makes use of a fallback if not otherwise specified.
    • By default the env.sh script's directory is assumed to hold the directory layout of the Deployment Area.
  • Each environment variable in the script points to a directory with a specific purpose that can be located outside of the suggested directory layout.
    • The DEP_ARCHIVE, DEP_BIN, DEP_CA, DEP_CONFIG, DEP_DESC, DEP_RELEASE, DEP_WORK environment variables point to directories explained with the directory layout.
    • Users can modify the directory layout at their will, for example the making the DEP_RELEASE environment variable indicate a mount point.
  • The location of the Java JDK is specified with the JAVA_HOME environment variable. Users have to adjusts the environment variable to the location of their Java JDK.
    • Typically Java LTS releases 11 and newer can be used. For Agent and Controller Java 1.8 can be used. For details see Which Java versions is JobScheduler available for?
    • The PATH environment variable is adjusted to point to the location of the bin sub-directory of Java.
  • Consider that environment variables have to be exported in order to be applied by subsequent calls to deployment scripts.

The Environment Script has to be sourced to make environment variables available to shell scripts in the Deployment Area:

The script has to be sourced, this means the call to the script is preceded by a dot and a space like this:

Run the env.sh Environment Script
. ./env.sh

Installer Wrapper Scripts: run_install_agent.sh, run_install_controller.sh, run_install_joc.sh

The Installer Wrapper Scripts parameterize calls to the Installer Scripts. The scripts are generated by the Packaging Script.

An example of a Wrapper Script for installation of an Agent using HTTPS connections looks like this:

Example for Installer Wrapper Script run_install_agent.sh
#!/bin/sh

set -e

DEP_CONFIG="/home/sos/js7.deploy/config"
DEP_BIN="/home/sos/js7.deploy/bin"
DEP_WORK_TEMP="/home/sos/js7.deploy/work/agents/agent_001"

mkdir -p "${DEP_WORK_TEMP}"

"${DEP_BIN}"/js7_install_agent.sh \
    --real-path-prefix="${DEP_WORK_TEMP}" \
    --release="2.5.0" \
    --tarball="/mnt/releases/scheduler_setups/2.5.0/js7_agent_unix.2.5.0.tar.gz" \
    --home="${DEP_WORK_TEMP}/opt/sos-berlin.com/js7/agent-primary" \
    --data="${DEP_WORK_TEMP}/var/sos-berlin.com/js7/agent-primary" \
    --http-port="localhost:31443" \
    --https-port="centostest-primary.sos:31443" \
    --controller-id="cluster" \
    --controller-primary-cert="${DEP_CONFIG}/controllers/instances/cluster.primary/config/centostest-primary.crt" \
    --controller-secondary-cert="${DEP_CONFIG}/controllers/instances/cluster.secondary/config/private/centostest-secondary.crt" \
    --keystore="${DEP_CONFIG}/agents/instances/agent_001/config/private/https-keystore.p12" \
    --keystore-password="jobscheduler" \
    --truststore="${DEP_CONFIG}/agents/instances/agent_001/config/private/https-truststore.p12" \
    --truststore-password="jobscheduler" \
    --deploy-dir="${DEP_CONFIG}/agents/templates/https/config" \
    --make-dirs


Explanation:

  • The parameterization is used from the Deployment Descriptor.
  • The --home and --data directories are prefixed with the directory of the work area.
    • The --real-path-prefix holds the work area directory. The Installer Script makes use of this information to conclude the effective directories as used for the target host.
  • Wrapper Scripts for Controllers and JOC Cockpit are similarly parameterized.

Deployment Scripts: run_deploy_agent.sh, run_deploy_controller.sh, run_deploy_joc.sh

The Deployment Scripts perform transfer of deployment tarballs to target machines and extracts the respective tarballs. The parameterization is created from the Deployment Descriptor.

Example for Deployment Script run_deploy_agent.sh
#!/bin/sh

set -e

scp -i "/home/sos/.ssh/sos_rsa" -P 22 "/home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz" sos@centostest-primary:"/tmp"
ssh -i "/home/sos/.ssh/sos_rsa" -p 22 -t -t -o LogLevel=QUIET sos@centostest-primary << 'EOF'
if [ $(id -u) -eq 0 ]
then
    use_sudo=""
else
    use_sudo="sudo"
fi

systemd_service_dir="/usr/lib/systemd/system"
systemd_service_name="js7_agent_31445.service"

...

mkdir -p "/opt/sos-berlin.com/js7"
cd "/opt/sos-berlin.com/js7" > /dev/null
test -e "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz" && gzip -c -d < "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.install.tar.gz" | tar -xf -
rc=$?
cd - > /dev/null

exit $rc
EOF


scp -i "/home/sos/.ssh/sos_rsa" -P 22 "/home/sos/js7.dep/archive/standalone-agent-http-2022-12-04/agents/agent_001/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz" sos@centostest-primary:"/tmp"
ssh -i "/home/sos/.ssh/sos_rsa" -p 22 -t -t -o LogLevel=QUIET sos@centostest-primary << 'EOF'
if [ $(id -u) -eq 0 ]
then
    use_sudo=""
else
    use_sudo="sudo"
fi

mkdir -p "/var/sos-berlin.com/js7"
cd "/var/sos-berlin.com/js7" > /dev/null
test -e "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz" && gzip -c -d < "/tmp/js7_deploy_agent_unix.agent_001.2.5.0.config.tar.gz" | tar -xf -
rc=$?
cd - > /dev/null

systemd_service_dir="/usr/lib/systemd/system"
systemd_service_name="js7_agent_31445.service"

...

if [ "${rc}" -eq 0 ]
then
    MakeService "/opt/sos-berlin.com/js7/agent-standalone/bin/agent_31445.service"
    rc=$?
fi

StartService

exit $rc
EOF


Explanation:

  • Authentication is performed by use of private/public key.
  • For each deployed tarball transfer and extraction are individually applied.
  • The example includes commands for pre- and post-processing to stop and to start an Agent from iJS7 - systemd Service Files for automated Startup and Shutdown with Unix Systems.
  • The example leaves out (...) a number of shell functions that are created to implement the MakeService, StopService and StartService operations.

Resources



  • No labels