Page History
Table of Contents |
---|
Introduction
- This article describes the build process for official Agent images.
- Users can build their own Docker container images for Agents .This article explains options how to create the Agent imageand adjust to their needs.
Build Environment
For the build environment the The following directory hierarchy is assumed for the build environment:
agent
build.sh
build
Dockerfile
entrypoint.sh
js7_install_start-agent.sh
config
The root directory agent
could can have any name. Consider Note that below the build script listed below will, by default will , use the directory name and release number to determine the resulting image name.
The build script build.sh
and Agent start script start-agent and entrypoint script entrypoint.sh
are explained described below.
Dockerfile
Download: Dockerfile
Container Docker images for JS7 Agents provided by SOS make use of the following Dockerfile:
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
FROM openjdk:8-jre-alpine LABEL maintainer="Software- und Organisations-Service GmbH" # BUILD SETTINGS# BUILD PRE-IMAGE FROM alpine:3.17 AS js7-pre-image # provide build arguments for release information ARG JS_RELEASE ARG JS_RELEASE_MAJOR # defaultimage user id has to match later run-time user id ARG JS_USER_ID=${UID:-1001} ARG JS_HTTP_PORT=${JS_HTTP_PORT:-4445} ARG JS_HTTPS_PORT=${JS_HTTPS_PORT:-4443""} ARG JS_JAVA_OPTIONS=${JS_JAVA_OPTIONS} # RUN-TIME SETTINGS add/copy installation tarball # JS7 JobScheduler ports and Java options ENV RUN_JS_HTTP_PORT=${RUN_JS_HTTP_PORT:-$JS_HTTP_PORT} ENV RUN_JS_HTTPS_PORT=${RUN_JS_HTTPS_PORT} ENV RUN_JS_JAVA_OPTIONS=${RUN_JS_JAVA_OPTIONS:-$JS_JAVA_OPTIONS} # PREPARATION # install process tools, net tools, bash RUN apk update && apk add --no-cache \ procps \ net-tools \ bash # setup working directory RUN mkdir -p /var/sos-berlin.com/js7 WORKDIR /var/sos-berlin.com/js7 # add/copy installation tarball # ADD https://download.sos-berlin.com/ADD https://download.sos-berlin.com/JobScheduler.${JS_RELEASE_MAJOR}/js7_agent_unix.${JS_RELEASE}.tar.gz /usr/local/src/ COPY js7_agent_unix.${JS_RELEASE}.tar.gz /usr/local/src/ # test installation tarball RUN test -e /usr/local/src/js7_agent_unix.${JS_RELEASE}.tar.gz # add/copy installer script # ADD https://download.sos-berlin.com/JobScheduler.${JS_RELEASE_MAJOR}/js7_install_agent_unix.${JS_RELEASE}.tar.gz.sh /usr/local/srcbin/ COPY js7_install_agent_unix.${JS_RELEASE}.tar.gz.sh /usr/local/srcbin/ # add start script COPY start-agent.sh /usr/local/bin/ # INSTALLATION # extract tarball # for JDK < 12, /dev/random does not provide sufficient entropy, see https://kb.sos-berlin.com/x/lIM3 # add jobscheduler user account and make it the owner of directories RUN adduser -u ${JS_USER_ID:-1001} --disabled-password --home /home/jobscheduler --no-create-home --shell /bin/bash jobscheduler jobscheduler && \ test -e RUN adduser -u ${JS_USER_ID} -G root --disabled-password --home /home/jobscheduler --shell /bin/bash jobscheduler && \ chmod +x /usr/local/bin/js7_install_agent.sh && \ /usr/local/bin/js7_install_agent.sh \ --home=/opt/sos-berlin.com/js7/agent \ --data=/var/sos-berlin.com/js7/agent \ --tarball=/usr/local/src/js7_agent_unix.${JS_RELEASE}.tar.gz && \ tar xfvz /usr/local/src/js7_agent_unix.${JS_RELEASE}.tar.gz -C /var/sos-berlin.com/js7 && \ --user=jobscheduler \ rm /usr/local/src/js7_agent_unix. --http-port=${JS_HTTP_PORT} \ --java-options="${JS_RELEASE}.tar.gz && JAVA_OPTIONS}" \ chown -R jobscheduler:jobscheduler /var/sos-berlin.com --make-dirs && \ chmodrm +x-f /usr/local/binsrc/start-agent.sh && \ sed -i 's/securerandom.source=file:\/dev\/random/securerandom.source=file:\/dev\/urandom/g' /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/java.security # CONFIGURATION # copy default configuration # COPY config/ /var/sos-berlin.com/js7/agent/var_$JS_HTTP_PORT/config/ # CODA #js7_agent_unix.${JS_RELEASE}.tar.gz # BUILD IMAGE FROM alpine:3.17 AS js7-image LABEL maintainer="Software- und Organisations-Service GmbH" # provide build arguments for release information ARG JS_RELEASE ARG JS_RELEASE_MAJOR # default user id has to match later run-time user, can be overwritten when running the container USER jobscheduler # allow incoming traffic to ports EXPOSE $RUN_ ARG JS_USER_ID=${UID:-1001} ARG JS_HTTP_PORT=${JS_HTTP_PORT $RUN_:-4445} ARG JS_HTTPS_PORT=${JS_HTTPS_PORT CMD ["sh","-c","/usr/local/bin/start-agent.sh --http-port=$RUN_JS_HTTP_PORT --https-port=$RUN_JS_HTTPS_PORT --java-options=\"$RUN_JS_JAVA_OPTIONS\""] |
Explanations:
- Line 1: The base image is OpenJDK Java 1.8 (Debian based). You can run Agents with newer Java releases, however, stick to Oracle, OpenJDK or AdoptOpenJDK as the source for your Java base image. Alternatively you can use your own base image and install Java 1.8 on top of this.
- Line 8 - 9: The release identification is injected by build arguments. This information is used to determine the tarball to be downloaded.
- Line 12 - 15: Defaults for the user id running the Agent inside the container as well as HTTP and HTTPS ports are provided. These values can be overwritten by providing the respective build arguments.
- Line 20 - 22: Environment variables are provided at run-time, not at build-time. They can be used to specify ports and Java options when running the container.
- Line 27 - 31: The image OS is updated and additional packages are installed (ps, netstat, bash, vi).
- Line 38 - 39: You can either download the Agent tarball directly from the SOS web site or you store the tarball with the build directory and copy from this location.
- Line 53: if a
config
folder is available in the build directory then its contents is copied to the respectiveconfig
folder in the image. This can be useful to create an image with individual settings in configuration files, see JS7 - Agent Configuration Items. Line 56: The
start-agent.sh
script is copied from the build directory to the image. Users can apply their own version of the start script. The start script used by SOS looks like this:Code Block language bash title Agent Start Script linenumbers true collapse true #!/bin/sh js_http_port="" js_https_port="" js_java_options="" for option in "$@" do case "$option" in --http-port=*) js_http_port=`echo "$option" | sed 's/--http-port=//'` ;; --https-port=*) js_https_port=`echo "$option" | sed 's/--https-port=//'` ;; --java-options=*) js_java_options=`echo "$option" | sed 's/--java-options=//'` ;; *) echo "unknown argument: $option" exit 1 ;; esac done js_args="" if [ ! "$js_http_port" = "" ] then js_args="$js_args --http-port=$js_http_port" fi if [ ! "$js_https_port" = "" ] then js_args="$js_args --https-port=$js_https_port" fi if [ ! "$js_java_options" = "" ] then js_args="$js_args --java-options=$js_java_options" fi echo "starting Agent: /var/sos-berlin.com/js7/agent/bin/agent.sh start $js_args" /var/sos-berlin.com/js7/agent/bin/agent.sh start $js_args && tail -f /dev/null
- Line 59 - 60: The user account
jobscheduler
is created and is assigned the user id and group id handed over by the respective build arguments. This translates to the fact that the account running the Agent inside the container and the account that starts the container are assigned the same user id and group id. This allows the account running the container to access any files created by the Agent in mounted volumes with identical permissions. - Line 69: The HTTP port and optionally the HTTPS port are exposed to the Docker host. Both ports can be forwarded by environment variables when running the container, overwriting the build-time values. This is relevant only if users want to use ports inside the container that are different from the default values. In most situations the default ports should be fine and are mapped to outside ports on the Docker host when starting the container.
- Line 71: The start script is executed and is dynamically parameterized from environment variables that are forwarded when starting the container.
Build Script
The build script offers a number of options to parameterize the Dockerfile:
:-""}
ARG JS_JAVA_OPTIONS=${JS_JAVA_OPTIONS}
# JS7 JobScheduler user id, ports and Java options
ENV RUN_JS_USER_ID=${RUN_JS_USER_ID:-1001}
ENV RUN_JS_HTTP_PORT=${RUN_JS_HTTP_PORT:-$JS_HTTP_PORT}
ENV RUN_JS_HTTPS_PORT=${RUN_JS_HTTPS_PORT}
ENV RUN_JS_JAVA_OPTIONS=${RUN_JS_JAVA_OPTIONS:-$JS_JAVA_OPTIONS}
COPY --from=js7-pre-image ["/opt/sos-berlin.com/js7", "/opt/sos-berlin.com/js7"]
COPY --from=js7-pre-image ["/var/sos-berlin.com/js7", "/var/sos-berlin.com/js7"]
# copy configuration
COPY config/ /var/sos-berlin.com/js7/agent/config/
# copy entrypoint script
COPY entrypoint.sh /usr/local/bin/
# install process tools, net tools, bash, openjdk
# for JDK < 12, /dev/random does not provide sufficient entropy, see https://kb.sos-berlin.com/x/lIM3
# make default user the owner of directories
RUN apk update && apk add --no-cache \
--repository=http://dl-cdn.alpinelinux.org/alpine/edge/community \
procps \
net-tools \
su-exec \
bash \
shadow \
openjdk11 && \
sed -i 's/securerandom.source=file:\/dev\/random/securerandom.source=file:\/dev\/urandom/g' /usr/lib/jvm/java-11-openjdk/conf/security/java.security && \
sed -i 's/jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \\/jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, TLSv1, TLSv1.1, \\/g' /usr/lib/jvm/java-11-openjdk/conf/security/java.security && \
adduser -u ${JS_USER_ID} -G root --disabled-password --home /home/jobscheduler --shell /bin/bash jobscheduler && \
chown -R jobscheduler:root /opt/sos-berlin.com /var/sos-berlin.com && \
chmod -R g=u /etc/passwd /opt/sos-berlin.com /var/sos-berlin.com && \
chmod +x /usr/local/bin/entrypoint.sh
# START
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] |
Explanation:
- The Dockerfile implements two stages to exclude installer files from the resulting image.
- Line 3: The base image is the current Alpine image at build-time.
- Line 6 - 8: The release identification is injected by build arguments. This information is used to determine the tarball to be downloaded or copied.
- Line 10 - 13: Defaults for the user id running the Agent inside the container as well as HTTP and HTTPS ports are provided. These values can be overwritten by providing the relevant build arguments.
- Line 16 - 17: Users can either download the Agent tarball directly from the SOS web site or store the tarball with the build directory and copy from this location.
Line 20: The tarball integrity is tested.
- Line 23 - 24: The Agent Installer Script is downloaded or copied, see JS7 - Unix Shell Installation Script - js7_install_agent.sh
- Line 27: The
jobscheduler
account is created. - Line 27 - 35: The Agent Installer Script is executed with arguments performing installation for the
jobscheduler
account. - Line 55 - 58: Environment variables are provided at run-time, not at build-time. They can be used to specify ports and Java options when running the container.
- Line 64: The
config
folder available in the build directory is copied to theconfig
sub-folder in the image. The parent foldervar_<port>
is determined from the HTTP port that the Agent is built for. This can be useful for creating an image with individual default settings in configuration files, see the JS7 - Agent Configuration Items article for more information. - Line 67: The
entrypoint.sh
script is copied from the build directory to the image. Users can apply their own version of the entrypoint script. The entrypoint script used by SOS looks like this: - Line 72 - 78: The image OS is updated and additional packages are installed (ps, netstat, bash).
- Line 79: The most recent Java 11 package available with Alpine is applied. Agents can be operated with newer Java releases. However, stick to Oracle, OpenJDK or AdoptOpenJDK as the source for your Java LTS release. Alternatively you can use your own base image and install Java on top of this. For details see Which Java versions is JobScheduler available for?
- Line 80: Java releases might make use of
/dev/random
for random number generation. This is a bottleneck as random number generation with this file is blocking. Instead/dev/urandom
should be used that implements non-blocking behavior. The change of the random file is applied to the Java security file. - Line 81: Users might want to disable certain TLS protocol versions or algorithms by applying changes to the Java security file.
- Line 82 - 84: The
jobscheduler
account is created and is assigned the user id handed over by the relevant build argument. This suggests that the account running the Agent inside the container and the account that starts the container are assigned the same user id. This allows the account running the container to access any files created by the Agent in mounted volumes with identical permissions.- Consider that the account is assigned the
root
group. For environments in which the entrypoint script is executed with an arbitrary non-root user id this allows access to files created by the Agent provided to any accounts that are assigned theroot
group. - Accordingly any files owned by the
jobscheduler
account are made accessible to theroot
group with similar user permissions. Read access to/etc/passwd
can be required in such environments. - For details see JS7 - Running Containers for User Accounts.
- Consider that the account is assigned the
- Line 85: The entrypoint script is made executable.
- Line 89: The entrypoint script is executed and is dynamically parameterized from environment variables when starting the container.
Entrypoint Script
Download: entrypoint.sh
The following entrypoint script is used to start Agent containers.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
#!/bin/bash
js_args=()
js_args_count=0
if [ -n "${RUN_JS_HTTP_PORT}" ] && [ ! "${RUN_JS_HTTP_PORT}" = ":" ]
then
js_args["${js_args_count}"]="--http-port=${RUN_JS_HTTP_PORT}"
js_args_count=$(( "${js_args_count}" + 1 ))
fi
if [ -n "${RUN_JS_HTTPS_PORT}" ] && [ ! "${RUN_JS_HTTPS_PORT}" = ":" ]
then
js_args["${js_args_count}"]="--https-port=${RUN_JS_HTTPS_PORT}"
js_args_count=$(( "${js_args_count}" + 1 ))
fi
if [ -n "${RUN_JS_JAVA_OPTIONS}" ]
then
js_args["${js_args_count}"]="--java-options=\"${RUN_JS_JAVA_OPTIONS}\""
js_args_count=$(( "${js_args_count}" + 1 ))
fi
if [ -n "${RUN_JS_JOB_JAVA_OPTIONS}" ]
then
js_args["${js_args_count}"]="--job-java-options=\"${RUN_JS_JOB_JAVA_OPTIONS}\""
js_args_count=$(( "${js_args_count}" + 1 ))
fi
# work directory will be created by container
if [ -d "/var/sos-berlin.com/js7/agent/work" ] && [ -w "/var/sos-berlin.com/js7/agent/work" ]
then
rm -f -r /var/sos-berlin.com/js7/agent/work
fi
JS_USER_ID=$(echo "${RUN_JS_USER_ID}" | cut -d ':' -f 1)
JS_GROUP_ID=$(echo "${RUN_JS_USER_ID}" | cut -d ':' -f 2)
JS_USER_ID=${JS_USER_ID:-$(id -u)}
JS_GROUP_ID=${JS_GROUP_ID:-$(id -g)}
BUILD_USER_ID=$(cat /etc/passwd | grep jobscheduler | cut -d ':' -f 4)
BUILD_GROUP_ID=$(cat /etc/group | grep jobscheduler | cut -d ':' -f 3)
if [ "$(id -u)" = "0" ]
then
if [ ! "${BUILD_USER_ID}" = "{$JS_USER_ID}" ]
then
echo "JS7 entrypoint script switchng ownership of image user id '${BUILD_USER_ID}' -> '${JS_USER_ID}'"
usermod -u "${JS_USER_ID}" jobscheduler
find /var/sos-berlin.com/ -user "${BUILD_USER_ID}" -exec chown -h jobscheduler {} \;
fi
if [ ! "${BUILD_GROUP_ID}" = "${JS_GROUP_ID}" ]
then
if grep -q "${JS_GROUP_ID}" /etc/group
then
groupmod -g "${JS_GROUP_ID}" jobscheduler
else
addgroup -g ${JS_GROUP_ID} -S jobscheduler
fi
echo "JS7 entrypoint script switchng ownership of image group id '${BUILD_GROUP_ID}' -> '${JS_GROUP_ID}'"
find /var/sos-berlin.com/ -group "${BUILD_GROUP_ID}" -exec chgrp -h jobscheduler {} \;
fi
echo "JS7 entrypoint script switching to user account 'jobscheduler' to run start script"
echo "JS7 entrypoint script starting Agent: exec su-exec ${JS_USER_ID}:${JS_GROUP_ID} /opt/sos-berlin.com/js7/agent/bin/agent_4445.sh start-docker" "${js_args[@]}"
exec su-exec "${JS_USER_ID}":"${JS_GROUP_ID}" /opt/sos-berlin.com/js7/agent/bin/agent_4445.sh start-docker "${js_args[@]}"
else
if [ "${BUILD_USER_ID}" = "${JS_USER_ID}" ]
then
if [ "$(id -u)" = "${JS_USER_ID}" ]
then
echo "JS7 entrypoint script running for user id '$(id -u)'"
else
echo "JS7 entrypoint script running for user id '$(id -u)' using user id '${JS_USER_ID}', group id '${JS_GROUP_ID}'"
echo "JS7 entrypoint script missing permission to switch user id and group id, consider to omit the 'docker run --user' option"
fi
else
echo "JS7 entrypoint script running for user id '$(id -u)' using image user id '${BUILD_USER_ID}' -> '${JS_USER_ID}', image group id '${BUILD_GROUP_ID}' -> '${JS_GROUP_ID}'"
fi
echo "JS7 entrypoint script starting Agent: exec sh -c /opt/sos-berlin.com/js7/agent/bin/agent_4445.sh start-docker ${js_args[*]}"
exec sh -c "/opt/sos-berlin.com/js7/agent/bin/agent_4445.sh start-docker ${js_args[*]}"
fi |
Build Script
The build script offers a number of options to parameterize the Dockerfile:
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
#!/bin/bash
set -e
SCRIPT_HOME=$(dirname "$0")
SCRIPT_HOME="`cd "${SCRIPT_HOME}" >/dev/null && pwd`"
SCRIPT_FOLDER="`basename $(dirname "$SCRIPT_HOME")`"
| ||||||||
Code Block | ||||||||
| ||||||||
#!/bin/bash set -e SCRIPT_HOME=$(dirname "$0") SCRIPT_HOME="`cd "${SCRIPT_HOME}" >/dev/null && pwd`" SCRIPT_FOLDER="`basename $(dirname "$SCRIPT_HOME")`" # ----- modify default settings ----- JS_RELEASE="2.0.0-SNAPSHOT" JS_REPOSITORY="sosberlin/js7" JS_IMAGE="$(basename "${SCRIPT_HOME}")-${JS_RELEASE//\./-}" JS_USER_ID="$UID" JS_NETWORK="js7" JS_HTTP_PORT="4445" JS_HTTPS_PORT="4443" JS_JAVA_OPTIONS="-Xmx500m" JS_BUILD_ARGS="" # ----- modify default settings ----- JS_RELEASE="2.5.0" JS_REPOSITORY="sosberlin/js7" JS_IMAGE="$(basename "${SCRIPT_HOME}")---- for option in "$@" do case "$option" in --release=*) JS_RELEASE=`echo "$option" | sed 's/--release=//'` ;;${JS_RELEASE//\./-}" JS_USER_ID="$UID" JS_HTTP_PORT="4445" JS_HTTPS_PORT= JS_JAVA_OPTIONS="-Xmx256m" JS_BUILD_ARGS= # ----- modify default settings ----- for option in "$@" do case "$option" in --repositoryrelease=*) JS_REPOSITORYRELEASE=`echo "$option" | sed 's/--repositoryrelease=//'` ;; --imagerepository=*) JS_IMAGEREPOSITORY=`echo "$option" | sed 's/--imagerepository=//'` ;; --user-idimage=*) JS_USER_IDIMAGE=`echo "$option" | sed 's/--user-idimage=//'` ;; --user-networkid=*) JS_NETWORKUSER_ID=`echo "$option" | sed 's/--user-networkid=//'` ;; --http-port=*) JS_HTTP_PORT=`echo "$option" | sed 's/--http-port=//'` ;; --https-port=*) JS_HTTPS_PORT=`echo "$option" | sed 's/--https-port=//'` ;; --java-options=*) JS_JAVA_OPTIONS=`echo "$option" | sed 's/--java-options=//'` ;; --build-args=*) JS_BUILD_ARGS=`echo "$option" | sed 's/--build-args=//'` ;; *) echo "unknown argument: $option" exit 1 ;; esac done set -x docker build --no-cache --rm \ --tag=$JS_REPOSITORY:$JS_IMAGE \ --file=$SCRIPT_HOME/build/Dockerfile \ --network=$JS_NETWORK/Dockerfile \ --build-arg="JS_RELEASE=$JS_RELEASE" \ --build-arg="JS_RELEASE_MAJOR=$(echo $JS_RELEASE | cut -d . -f 1,2)" \ --build-arg="JS_USER_ID=$JS_USER_ID" \ --build-arg="JS_HTTP_PORT=$JS_HTTP_PORT" \ --build-arg="JS_HTTPS_PORT=$JS_HTTPS_PORT" \ --build-arg="JS_JAVA_OPTIONS=$JS_JAVA_OPTIONS" \ $JS_BUILD_ARGS $SCRIPT_HOME/build set +x |
ExplanationsExplanation:
- Line 12 - 2322: Default values are specified that are used if no command line arguments are provided. This includes values for:
- the release number: adjust this value to a current the release of JS7 that you want to build an Agent for.
- the repository which by default is
sosberlin:js7
. - the image name is determined from the current folder name and the release number.
- the user id is by default the user id of the user running the build script.
- the Docker network: the build script assumes a Docker network to be used for which a name is specified.the HTTP port and HTTPS port: if the respective port is not specified then the Agent will not listen to a port for the respective associated protocol. You can for example disable the HTTP protocol by specifying an empty value. The default ports should be fine as they are mapped by the run script to outside ports on the Docker container's host. However, you can modify ports as you likerequired.
- Java options: typically you would specify default values e.g. for Java memory consumption. The Java options can be overwritten by the run script when starting the container. However, however, you might want to create your own image with adjusted default values.
- Line 28 27 - 5350: The above options can be overwritten by command line arguments like this:
Code Block language bash title Running the Build Script with Arguments linenumbers true ./build.sh --network=js --http-port=14445 --https-port=14443 --java-options="-Xmx1G"
- Line 57 54 - 6763: The effective
docker build
command is executed with arguments. The Dockerfile is assumed to be located with in thebuild
sub-directory of the current directory.
...