Introduction

Installation, updates and upgrades are performed using the .tar.gz/.zip installer archives provided for the initial installation of newer releases.

For JS7 - Secure Operation the connections between JS7 JOC Cockpit, Controller and Agents are secured by TLS/SSL certificates.

• Users create a keystore that holds the private key and TLS/SSL certificate for the respective server. Users create a truststore that holds the Root CA Certificate and optionally Intermediate CA Certificate(s).
• The process of creating and updating keystores and truststores
• JS7 - Installation instructions apply.
• For environments with a larger number of JOC Cockpit instances the update, upgrade and patch processes can be automated in a number of ways:
  • Users can use their preferred tools such as Ansible®, Puppet®, Chef®.
  • Users can apply the JOC Cockpit Installer Certificate Management Script that is described in this article either standalone or in combination with such tools.

Security

Secure rollout of JS7 components is critical. It is therefore recommended that the solution described here is adjusted to suit specific security needs.

• Rollout of JS7 JOC Cockpit is considered critical as the software allows jobs to be executed on a larger number of servers.
  • Attention should be paid to the integrity of the sources for JS7 component downloads.
  • This includes intermediate devices on which JS7 software installers are stored in a user's environment.
  • One option is to run the JOC Cockpit Installer Script from sudo and to use the digest functionality that compares the script to a hash value stored with the sudoers file.
• The solution for updating, upgrading and patching the JS7 JOC Cockpit is based on shell scripting by design
  • to provide readability and to rely only on OS commands,
  • to prohibit the use of any 3rd-party components and additional dependencies that require code to be executed on the machines that run the JOC Cockpit.
• The JOC Cockpit Installer Script can be integrated in a number of ways:
  • by running one's own SSH script on top of the JOC Cockpit Installer Script,
  • by use of tools such as Ansible®, Puppet® that make use of an SSH Client,
  • by use of JS7 workflow automation as explained below.
    • It is recommended that a separate Standalone Controller and Agent are used for rollout purposes.
    • Access to the Controller and Agent for rollout should be securely managed.

Certificate Store Management Script

The Certificate Store Management Script is provided for download and can be used to automate the creation of keystores and truststores.

• The script is available for Linux, MacOS® and AIX® using bash, dash, ksh, and zsh shells.
• The script terminates with exit code 0 to signal success, with exit code 1 for command line argument errors and with exit code 2 for non-recoverable errors.
• The script is intended as a baseline example for customization by JS7 users and by SOS within the scope of professional services.

Download

Find the Certificate Store Management Script for download from JS7 - Download.

Usage

Invoking the Certificate Store Management Script without arguments displays the usage clause:

Usage: js7_create_certificate_store.sh [Options] [Switches]

  Options:
    --keystore=|keystore-file=<file>      | required: path to keystore file in PKCS12 format
    --truststore=|truststore-file=<file>  | optional: path to truststore file in PKCS12 format
    --key=<private-key-file>              | required: path to private key file in .pem format
    --cert=|certificate=<cert-file>       | required: path to certificate file in .pem format
    --alias=<alias-name>                  | required: alias name for keystore entry
    --password=<password>                 | required: password for certificate store
    --ca-bundle=<ca-bundle-file>          | optional: path to CA Bundle certificate file in .pem format
    --ca-root=<ca-root-file>              | optional: path to CA Root certificate file in .pem format
    --ca-intermediate=<ca-file[,ca-file]> | optional: path to CA Intermediate certificate file in .pem format
    --backup-dir=<directory>              | optional: backup directory for existing certificate stores
    --log-dir=<directory>                 | optional: log directory for log output of this script
  Switches:
    -h | --help                           | displays usage
    --chain                               | add certificate chain to keystore
    --show-logs                           | shows log output of the script
    --make-dirs                           | creates the specified directories if they do not exist

Explanation:

• Options
  • --keystore
    • Specifies the path to a PKCS12 keystore file that should hold the private key and certificate for HTTPS connections to JS7 components.
    • Users are free to specify any file name, typically the name https-keystore.p12 is used.
    • Further settings in the ssl.ini file such as the keystore password have to be deployed from a copy of the file using the --ini option.
    • Assigning a keystore for HTTPS connections disables HTTP access and enables HTTPS access only to JOC Cockpit. The same port is alternatively used for HTTP and HTTPS connections.
  • --truststore-file
    • Specifies the path to a PKCS12 truststore file that holds the certificate(s) for HTTPS connections from JOC Cockpit to a Controller instance, LDAP server etc.
    • Users are free to specify any file name, typically the name https-truststore.p12 is used. The truststore file will be copied to the <home>/jetty_base/resources/joc directory.
    • If a truststore file is made available then the JOC Cockpit's <home>/jetty_base/start.d/ssl.ini file has to hold a reference to the truststore location and optionally the truststore password. It is therefore recommended to use the --ini option to deploy an individual ssl.ini file. The following settings are automatically updated in the ssl.ini file:
      • jetty.sslContext.trustStorePath: specifies the path to the truststore relative to the <home>/jetty_base/resources/joc directory.
    • Further settings in the ssl.ini file such as the truststore password have to be deployed from a copy of the file using the --ini option.
  • --truststore
    
    • The JOC Cockpit installer is used with the joc_install.xml response file. This file is available after extraction of the installer tarball and specifies options for installation of the JOC Cockpit. The file is applied when invoking the installer by ./setup.sh -u joc_install.xml, see JS7 - JOC Cockpit - Headless Installation on Linux and Windows.
    • Users should keep their copy of the response file and specify the path with this command line option. Response files can be re-used within the same minor release of the JOC Cockpit, for example when updating from release 2.3.1 to 2.2.4. When updating, for example, from release 2.2.x to 2.3.x it is recommended a check is carride out from the installer tarball if a newer version of the file is available.
    • Users should note that the response file can hold references to a license file and to a JDBC Driver .jar file. The JOC Cockpit setup is executed from the directories specified with the --setup-dir option. Paths can be used relative to this directory.
  • --key
    • Specifies the path to the file that holds the private key that should be added to the keystore.
    • This argument is required if a keystore should be created, see --keystore.
  • --cert
    • Specifies the path to the file that holds the certificate that should be added to the keystore.
    • This argument is required if a keystore should be created, see --keystore.
  • --alias
    • Specifies the alias name of the private key and certificate entry in the keystore.
    • This argument is required if a keystore should be created, see --keystore.
  • --password
    • Specifies the password that protects the keystore and truststore.
    • Passwords for certificate stores are not intended to improve security but to prevent users from shooting themselves in their foot. For a keystore the password protects read and write access, for a truststore the password protects write access.
    • This argument is required if a keystore or truststore should be created, see --keystore, --truststore.
  • ca-bundle
    • Specifies the path to a CA Bundle file that holds the CA Intermediate certificate(s) and Root CA certificate in the indicated sequence.
    • This argument is required if a keystore should be created and the --chain switch is used. In addition this argument is required if a truststore should be created an no Root CA certificate is specified, see --ca-root.
  • ca-root
    • Specifies the path to a CA Root Certificate file.
    • This argument is required if no CA Bundle file is specified, see --ca-bundle, and a keystore should be created with the --chain argument. In addition this argument is required if a truststore should be created using the --truststore option and no CA Bundle file is specified, see --ca-bundle option.
  • ca-intermediate
    • Specifies the path to one or more CA Intermediate Certificate files.
    • If more than one file is specified then file names have to be separated by comma, for example --ca-intermediate="./certs/intermediate-ca-1.crt,./certs/intermediate-ca-2.crt".
    • This argument is required if no CA Bundle file is specified, see --ca-bundle, and a keystore should be created with the --chain argument.
  • --backup-dir
    • If a backup directory is specified then an existing keystore and truststore will be added to a .tar.gz file in this directory.
    • File names are created according to the pattern: backup_js7_certificate_store.<hostname>.<yyyy>-<MM>-<dd>T<hh>-<mm>-<ss>.tar.gz
    • For example: backup_js7_certificate_store.centostest_primary.2022-03-19T20-50-45.tar.gz
  • --log-dir
    • If a log directory is specified then the installer script logs information about processing steps to a log file in this directory.
    • File names are created like this: certificate_store._js7.<hostname>.<yyyy>-<MM>-<dd>T<hh>-<mm>-<ss>.log
    • For example: certificate_store_js7.centostest_primary.2022-03-19T20-50-45.log
• Switches
  • -h | --help
    • Displays usage.
  • --chain
    • Stops a running JOC Cockpit before installation and starts the JOC Cockpit after installation using the JOC Cockpit's Start Script. This switch can be used with the --kill switch to control the way how JOC
  • --show-logs
    • Displays the log output created by the script.
  • --make-dirs
    • If directories are missing that are indicated with the --keystore, --truststore, --backup-dir or --log-dir options then they will be created.
• Exit Codes
  • 1: argument errors
  • 2: non-recoverable errors
  • 3: this exit code is returned when used with the --restart switch and if it cannot be identified if a JOC Cockpit instance is running
  • 4: this exit code is returned if no --tarball option is used and download of the tarball reports errors
  • 5: this exit code is returned when used with the --restart switch and if the JOC Cockpit instance cannot be started
  • 6: this exit code is returned when used with the --restart switch and if the JOC Cockpit instance cannot be stopped
  • 7: this exit code indicates that the JOC Cockpit installation has failed
  • 8: this exit code indicates failure of the JOC Cockpit installation from logs

Examples

The following examples represent typical use cases. Users should consider to specify current releases, see JS7 - Download.

Install or Update from Download

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --release=2.3.1 \
    --make-dirs

# downloads the JOC Cockpit release indicated and extracts the installer tarball to the specified JOC Cockpit setup directory 
# the setup directory is created if it does not exist and the indicated response file for setup options is used

Install or Update from Download with Commercial License

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --release=2.3.1 \
    --license-key=/home/sos/example.pem \
    --make-dirs

# downloads the JOC Cockpit release indicated and extracts the installer tarball to the specified JOC Cockpit setup directory 
# the setup directory is created if it does not exist and the indicated response file for setup options is used
# installs the license key file and downloads the binary file for licensed code to enable cluster operations

Install or Update from Tarball

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gz

# extracts the tarball indicated to the specified JOC Cockpit setup directory

Install or Update from Tarball with Commercial License

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gz \
    --license-key=/home/sos/example.pem \
    --license-bin=/mnt/releases/js7/js7-license.jar

# extracts the tarball indicated to the specified JOC Cockpit setup directory
# installs the license key file and binary file for licensed code to enable cluster operations

Install or Update and Stop/Start using systemd

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gz \
    --exec-start="sudo systemctl start js7_joc" \
    --exec-stop="sudo systemctl stop js7_joc"

# extracts the tarball indicated to the specified JOC Cockpit setup directory 
# the JOC Cockpit is stopped and started using systemd commands

Install or Update and Restart

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gz \
    --restart

# extracts the tarball indicated to the specified JOC Cockpit setup directory 
# the JOC Cockpit is stopped and started from its own instance start script

Install or Update with Return Values

retval=/tmp/js7_install_joc.$$.tmp

./js7_install_joc.sh \
    --setup-dir=/home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xml \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gz \
    --backup-dir=/tmp/backups \
    --log-dir=/tmp/logs \
    --return-values=$retval \
    --restart

log_file=$(cat $retval | grep "log_file" | cut -d'=' -f2)
backup_file=$(cat $retval | grep "backup_file" | cut -d'=' -f2)

# extracts the tarball indicated to the specified JOC Cockpit setup directory, creates a log file and a backup file
# return values include the path to the log file and to the backup file

Install or Update with Fallback

Certificate Management Script

The Certificate Management Script is provided for download and can be used to automate the creation of keystores and truststores.

• The script is available for Linux, MacOS® and AIX® using bash, dash, ksh, and zsh shells.
• The script makes use of OpenSSL and the Java keytool available from a JRE/JDK.
• The script terminates with exit code 0 to signal success, with exit code 1 for command line argument errors and with other exit codes for non-recoverable errors.
• The script is intended as a baseline example for customization by JS7 users and by SOS within the scope of professional services.

Download

Find the Certificate Management Script for download from JS7 - Download.

Usage

Invoking the Certificate Management Script without arguments displays the usage clause:

Usage: js7_create_certificate_store.sh [Options] [Switches]

  Options:
    --keystore=<file>                     | required: path to keystore file in PKCS12 format
    --truststore==<file>                  | optional: path to truststore file in PKCS12 format
    --key=<private-key-file>              | required: path to private key file in .pem format
    --cert=<cert-file>                    | required: path to certificate file in .pem format
    --alias=<alias-name>                  | required: alias name for keystore entry
    --password=<password>                 | required: password for certificate store
    --ca-cert=<ca-cert-bundle-file>       | optional: path to CA Certificate Bundle file in .pem format
    --ca-root=<ca-root-file>              | optional: path to Root CA Certificate file in .pem format
    --ca-intermediate=<ca-file[,ca-file]> | optional: paths to Intermediate CA Certificate files in .pem format
    --backup-dir=<directory>              | optional: backup directory for existing certificate stores
    --log-dir=<directory>                 | optional: log directory for log output of this script
  Switches:
    -h | --help                           | displays usage
    --chain                               | add certificate chain to keystore
    --show-logs                           | shows log output of the script
    --make-dirs                           | creates the specified directories if they do not exist

Explanation:

• Options
  • --keystore
    • Specifies the path to a PKCS12 keystore file that should be created. The keystore holds the private key and server certificate for HTTPS connections to JS7 components.
    • Users are free to specify any file name, typically the name https-keystore.p12 is used.
    • Further settings in JS7 configuration files such as the keystore location and password have to be considered.
  • --truststore
    • Specifies the path to a PKCS12 truststore file that should be created. The truststore holds the certificate(s) for HTTPS connections from JS7 components.
    • Users are free to specify any file name, typically the name https-truststore.p12 is used.
    • Further settings in JS7 configuration files such as the truststore location and password have to be considered.
  • --key
    • Specifies the path to the file that holds the private key that should be added to the keystore.
    • This argument is required should a keystore be created, see --keystore.
  • --cert
    • Specifies the path to the file that holds the server/client authentication certificate that should be added to the keystore.
    • This argument is required should a keystore be created, see --keystore.
  • --alias
    • Specifies the alias name of the entry in the keystore that holds the private key and certificate.
    • This argument is required should a keystore be created, see --keystore.
  • --password
    • Specifies the password that protects the keystore and truststore.
    • Passwords for certificate stores are not intended to improve security but to prevent users from shooting themselves in their foot. For a keystore the password protects read and write access, for a truststore the password protects write access.
    • This argument is required should a keystore or truststore be created, see --keystore, --truststore.
  • ca-cert
    • Specifies the path to a CA Certificate Bundle file that holds the Intermediate Ca Certificate(s) and Root CA Certificate in the indicated sequence.
    • This argument is required should a keystore be created and the --chain switch be used. In addition this argument is required should a truststore be created and no Root CA Certificate be specified, see --ca-root.
  • ca-root
    • Specifies the path to a CA Root Certificate file.
    • This argument is required if no CA Certificate Bundle file is specified, see --ca-cert, and should a keystore be created with the --chain argument. In addition this argument is required should a truststore be created using the --truststore option and no CA Certificate Bundle file is specified, see --ca-cert option.
  • ca-intermediate
    • Specifies the path to one or more Intermediate CA Certificate files.
    • If more than one file is specified then file names have to be separated by comma, for example --ca-intermediate="./certs/intermediate-ca-1.crt,./certs/intermediate-ca-2.crt".
    • This argument is required if no CA Certificate Bundle file is specified, see --ca-cert, and should a keystore be created using the --chain switch.
  • --backup-dir
    • If a backup directory is specified then an existing keystore and truststore will be added to a .tar.gz file in this directory.
    • File names are created according to the pattern: backup_js7_<keystore|truststore>.<hostname>.<yyyy>-<MM>-<dd>T<hh>-<mm>-<ss>.tar.gz
    • For example: backup_js7_keystore.centostest_primary.2022-03-19T20-50-45.tar.gz
  • --log-dir
    • If a log directory is specified then the installer script logs information about processing steps to a log file in this directory.
    • File names are created like this: certificate_store_js7.<hostname>.<yyyy>-<MM>-<dd>T<hh>-<mm>-<ss>.log
    • For example: certificate_store_js7.centostest_primary.2022-03-19T20-50-45.log
• Switches
  • -h | --help
    • Displays usage.
  • --chain
    • Adds the certificate chain to the keystore if the --keystore option is used. In most situations this is not required as the certificate chain is made available with the truststore. The certificate chain will be created from the CA Certificate Bundle file, see --ca-cert, or from individual CA Certificates, see --ca-root, --ca-intermediate.
  • --show-logs
    • Displays the log output created by the script.
  • --make-dirs
    • If directories are missing that are indicated with the --keystore, --truststore, --backup-dir or --log-dir options then they will be created.
• Exit Codes
  • 1: argument errors, OpenSLL errors
  • 2: this exit code is returned in case of keystore/truststore verification errors.

Examples

The following examples represent typical use cases. Users should consider to specify current releases, see JS7 - Download.

Create Keystore and Truststore

./js7_create_certificate_store

retval=/tmp/js7_install_joc.$$.tmp

./js7_install_joc.sh \
    --setup-dirkeystore=./home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xmlcerts/https-keystore.p12 \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gztruststore=./certs/https-truststore.p12 \
    --backup-dirkey=./tmpcerts/backupsapmaccs.key \
    --log-dircert=./tmpcerts/logsapmaccs.crt \
    --return-valuesalias=$retvalapmaccs \
    --restart \
    --show-logs \
    --make-dirs \
 || ( backup=$(cat $retval | grep "backup_file" | cut -d'=' -f2) \
      && ( test -e "$backup" ) && \
      ./js7_install_joc.sh \
          --setup-dir=/home/sos/joc.setuppassword=jobscheduler \
    --ca-cert=./certs/ca-bundle.crt

# creates a keystore from the key and certificate
# creates a truststore from the CA Certificate Bundle
# the keystore and truststore will be protected with the given password

Create Keystore

./js7_create_certificate_store.sh \
    --keystore=./certs/https-keystore.p12 \
    --key=./certs/apmaccs.key \
    --cert=./certs/apmaccs.crt \
      --alias=apmaccs \
    --setup-response=/home/sos/joc.response/joc_install.xml \
       password=jobscheduler

# creates a keystore from the key and certificate
# the keystore will be protected with the given password

Create Keystore with Certificate Chain

./js7_create_certificate_store.sh \
    --keystore=./certs/https-keystore.p12   --tarball=$backup \
          --log-dir=/tmp/logs \
      --key=./certs/apmaccs.key \
    --restartcert=./certs/apmaccs.crt \
          --show-logs )

log_file=$(cat $retval | grep "log_file" | cut -d'=' -f2)
backup_file=$(cat $retval | grep "backup_file" | cut -d'=' -f2)

# extracts the tarball indicated to the specified JOC Cockpit setup directory, creates a backup file and a log file and restarts JOC Cockpit
# should installation fail then the installation from the backup file will be reverted to

...

--alias=apmaccs \
    --password=jobscheduler \
    --ca-cert=./certs/ca-bundle.crt \
    --chain

# creates a keystore from the key and certificate
# adds the certificate chain from the CA Certificate Bundle to the keystore
# the keystore will be protected with the given password

Create Truststore from CA Certificate Bundle

./js7_create_installcertificate_jocstore.sh \
    --setup-dirtruststore=./home/sos/joc.setup \
    --setup-response=/home/sos/joc.response/joc_install.xmlcerts/https-truststore.p12 \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.tar.gzpassword=jobscheduler \
    --ca-inicert="./joc.config/http.ini,./joc.config/https.ini,./joc.config/ssl.ini" \
    --http-port=4446certs/ca-bundle.crt
 
# creates a truststore from the CA Certificate Bundle
# the truststore will be protected with the given password

Create Truststore from CA Certificates

./js7_create_certificate_store.sh \
    --keystore-filetruststore=./joc.configcerts/https-keystoretruststore.p12 \
    --truststore-file=./joc.config/https-truststore.p12password=jobscheduler \
    ---make-dirsca-root=./certs/root-ca.crt \
    ---user \
    --preserve-envca-intermediate=./certs/sos.intermediate-ca-1.crt,./certs/sos.intermediate-ca-2.crt

# extractscreates thea tarballtruststore indicated tofrom the specifiedIntermediate JOCCA CockpitCertificates setupand directory
#the deploysRoot Jetty SSL configuration files that hold references to keystore and truststore
# deploys keystore and truststore files

...

CA Certficate
# the truststore will be protected with the given password

Create Keystore with Logs and Backups

./js7_create_certificate_store.sh./js7_install_joc.sh \
    --home=/home/sos/joc \
    --release=2.3.1 \
    --patchkeystore=low.patch-1

# downloads the patch indicated and extracts the tarball to the specified JOC Cockpit home directory
# the patch is stored in the JOC Cockpit's jetty_base/webapps/joc/WEB-INF/classes sub-directory

Patch from Tarball

./js7_install_joc.sh/certs/https-keystore.p12 \
    --key=./certs/apmaccs.key \
    --cert=./certs/apmaccs.crt \
    --home=/home/sos/jocalias=apmaccs \
    --tarball=/mnt/releases/js7/js7_joc_linux.2.3.1.low.patch-1.tar.gzpassword=jobscheduler \
    --log-dir=./logs \
    --patchbackup-dir=low.patch-1/backup

# extracts creates a keystore from the key patchand tarballcertificate
# indicatedcreates toa thelog specifiedfile JOCin Cockpitthe homeindicated directory
# thecreates patcha isbackup storedfile toof the JOC Cockpit's jetty_base/webapps/joc/WEB-INF/classes sub-keystore in the indicated directory

Automation

The JOC Cockpit Installer The Certificate Management Script can be executed from a job for automated updating and upgrading of JS7 JOC Cockpit instances.The steps for automation are similar to updating and upgrading JS7 Agents. You will find instructions for setting up workflow automation from the JS7 creation of keystores and truststores.

Further Resources

• JS7 - Automated Update of

...

• JOC Cockpit
• JS7 - Automated Update of Controller
• JS7 - Automated Update of Agent

...