Introduction

FEATURE AVAILABILITY STARTING FROM RELEASE 2.4.0

Log files include data which is considered sensitive by a number of users. This includes:

Hostnames, IP Addresses, Ports
Accounts
Database Names

There are situations where log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.

JS7 allows logs to be anonymized by replacing sensitive data with placeholders:

Example for Log Output holding Host names and Ports

2022-05-14T11:27:27,026 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-seconda
2022-05-14T11:27:27,829 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443)
2022-05-14T11:27:28,526 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:28,527 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:31,343 INFO  JControllerProxy-42  c.s.j.c.p.ProxyContext                       - 'standalone' (https://controller-2-0-standalone:4443): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO  JControllerProxy-41  c.s.j.c.p.ProxyContext                       - 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443): ProxyCoupled(1652520420689258)

Example for anonymized Log Output

2022-05-14T11:27:27,026 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:27,829 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:28,526 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:28,527 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:31,343 INFO  JControllerProxy-42  c.s.j.c.p.ProxyContext                       - 'standalone' (https://<host>:<port>): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO  JControllerProxy-41  c.s.j.c.p.ProxyContext                       - 'testsuite' cluster (https://<host>:<port>): ProxyCoupled(1652520420689258)

Log Anonymizer Script

Location

Anonymization is available from a Java class and is invoked by a shell script available at the following default locations:

Location of Log Anonymizer Script for Unix

# JOC Cockpit
/opt/sos-berlin.com/js7/joc/jetty/bin/anonymize-logs.sh

# Controller
/opt/sos-berlin.com/js7/controller/bin/anonymize-logs.sh

# Agent
/opt/sos-berlin.com/js7/agent/bin/anonymize-logs.sh

Location of Log Anonymizer Script for Windows

@rem JOC Cockpit
C:\Program Files\sos-berlin.com\js7\joc\jetty\bin\anonymize-logs.cmd

@rem Controller
C:\Program Files\sos-berlin.com\js7\controller\bin\anonymize-logs.cmd

@rem Agent
C:\Program Files\sos-berlin.com\js7\agent\bin\anonymize-logs.cmd

Usage

Invocation of Log Anonymizer Script

Usage: anonymize-logs.sh [Options]

  Options:
    -l | --log-file=<log-file>       | optional: location of log files to be anonymized; a single file, directory or wildcards can be specified;
                                                 the argument can occur any number of times
    -o | --output-dir=<directory>    | optional: output directory for anonymized log files
    -r | --rules-file=<rules-file>   | optional: path to a YAML file holding rules for anonymization; by default built-in rules will be applied
    -e | --export-rules=<rules-file> | optional: path to a YAML file to which built-in rules will be exported

Explanation:

Options
- -l | --log-file=<log-file>: Specifies the location of the log file(s) to be anonymized.
  - This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified if all included files are to be anonymized.
  - Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
  - Anonymized log file names are prefixed with the string: anonymized-
  - By default anonymized log files are stored in the directory in which original log files are found.
- -o | --output-dir=<directory>: Optionally specifies the output directory in which anonymized log files are to be stored. If this argument is omitted then anonymized log files are stored in their original directory.
- -r | --rules-file=<rules-file>: Optionally specifies the location of a file in YAML format that holds the rules to be applied for anonymization.
- -e | --export-rules=<rules-file>: Optionally specifies the location of a file in YAML format to which the built-in rules for anonymization will be exported.

Rules

Rules include specifying regular expressions for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.

Users can export the built-in rules to a file to verify available rules and expressions.
Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended that individual files are validated as being YAML compliant.

Example for built-in Rules

rules:
- item: url-component
  search: ://(.*):(\d{2,5})
  replace:
  - <host>
  - <port>
- item: ip-address
  search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))
  replace:
  - <ip-address>
- item: host-install
  search: host[\s]*=[\s]*(.*)
  replace:
  - <host>

Explanation:

Each item specifies an expression to search for and to replace.
The search expression makes use of capturing groups specified by ( ... ).
The replace placeholder specifies a number of strings that replace the content of related capturing groups.

Examples

The following examples illustrate typical use cases.

Example for Invocation of Log Anonymizer Script

anonymize-logs.sh --log-file=/var/sos-berlin.com/js7/controller/logs/controller.log

Explanation:

Creates an anonymized version of the log file in the same directory with the name: anonymized-controller.log

Example for Invocation of Log Anonymizer Script for a number of Log Files

anonymize-logs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log  \
                  --log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log \
                  --output-dir=/tmp/logs

Explanation:

Creates anonymized versions of the indicated log files in the given output directory:
- /tmp/logs/anonymized-joc.log
- /tmp/logs/anonymized-joc-debug.log

Example for Invocation of Log Anonymizer Script with Wildcards for Log Files

anonymize-logs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc*  \
                  --output-dir=/tmp/logs

Explanation:

Creates anonymized versions of the log files indicated from the given directory. The wildcard specifies any log files carrying a name that starts with joc will be used.
The wildcard includes consideration of plain text log files with the .log file extension and also consideration of compressed log files with a .gz file extension.
Anonymized log files are stored in the output directory.

Example for Export of Rules to a File

anonymize-logs.sh --export-rules=/tmp/rules.yaml

Explanation:

The build-in rules are exported to a file in YAML format.
This file can be used to adjust rules and can be applied for anonymization later on.

Example for Invocation of Log Anonymizer Script with individual Rules from a File

anonymize-logs.sh --log-file=/var/sos-berlin.com/js7/agent_4445/logs/*.log \
                  --output-dir=/tmp/logs \
                  --rules-file=/tmp/rules.yaml

Explanation:

Creates anonymized versions of all log files in the indicated directory and stores them in the output directory.
A YAML file with individual rules is applied.

Running Log Anonymizer outside of JS7

Users who wish to run the Log Anonymizer outside of a JS7 installation can:

copy a number of Java related files as available from the Controller or Agent installation,
create a shell script to invoke the Log Anonymizer.

Java related Files required by Log Anonymizer

The following files can be found in a Controller or Agent installation for Unix or Windows.

Note that version numbers of binary files can change over time. The examples below assume Java related files as provided with Release 2.4.0.
Users can copy the following Java related files from the ./lib/3rd-party, ./lib/sos, ./lib/stdout directories to the location where they want to use Log Anonymizer:

Java binary files required by Log Anonymizer

./lib/sos/sos-commons-util-2.4.0.jar
./lib/stdout/log4j2.xml
./lib/3rd-party/org.apache.logging.log4j.log4j-api-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-core-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
./lib/3rd-party/org.slf4j.slf4j-api-1.7.36.jar
./lib/3rd-party/snakeyaml-1.30.jar

Shell Script to run Log Anonymizer

An individual script to invoke Log Anonymizer can look like this:

Example for Invocation of Log Anonymizer from an individual Script on Unix

#!/bin/sh

JAVA_BIN="${JAVA_HOME}/bin/java"
test -x "${JAVA_BIN}" || JAVA_BIN="java"

script_dir="$(echo $(dirname "$0") | cd > /dev/null && pwd)"
cp="-classpath "${script_dir}/lib/sos/*:${script_dir}/lib/3rd-party/*:${script_dir}/lib/stdout""

"${JAVA_BIN}" ${JAVA_OPTIONS} ${cp} com.sos.commons.util.loganonymizer.SOSLogAnonymizer $*

Example for Invocation of Log Anonymizer from an individual Script on Windows

@echo off

set JAVA_BIN=java.exe
if exist "%JAVA_HOME%\bin\java.exe" set JAVA_BIN=%JAVA_HOME%\bin\java.exe

set script_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\stdout"

"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %*

Explanation:

The directory hierarchy is assumed like this:

anonymize-logs.sh | .cmd
- lib
  - sos

- - - sos-commons-util-2.4.0.jar
  - stdout
    - log4j2.xml
  - 3rd-party
    - org.apache.logging.log4j.log4j-api-2.17.2.jar
    - org.apache.logging.log4j.log4j-core-2.17.2.jar
    - org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
    - org.slf4j.slf4j-api-1.7.36.jar
    - snakeyaml-1.30.jar

Space shortcuts

Page tree

Introduction

Log Anonymizer Script

Location

Usage

Rules

Examples

Running Log Anonymizer outside of JS7

Java related Files required by Log Anonymizer

Shell Script to run Log Anonymizer

Space shortcuts

Page tree

JS7 - Log Anonymization

Introduction

Log Anonymizer Script

Location

Usage

Rules

Examples

Running Log Anonymizer outside of JS7

Java related Files required by Log Anonymizer

Shell Script to run Log Anonymizer