Introduction
| Display feature availability |
|---|
|
Log files include data that are which is considered sensitive by a number of users. This includes:
- Hostnames, IP Addresses, Ports
- Accounts
- Database Names
There are situations when where log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.
JS7 offers to anonymize logs allows logs to be anonymized by replacing sensitive data with placeholders:
| Code Block |
|---|
| title | Example for Log Output holding Host names and Ports |
|---|
|
2022-05-14T11:27:27,026 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-seconda
2022-05-14T11:27:27,829 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443)
2022-05-14T11:27:28,526 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:28,527 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:31,343 INFO JControllerProxy-42 c.s.j.c.p.ProxyContext - 'standalone' (https://controller-2-0-standalone:4443): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO JControllerProxy-41 c.s.j.c.p.ProxyContext - 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443): ProxyCoupled(1652520420689258) |
| Code Block |
|---|
| title | Example for anonymized Log Output |
|---|
|
2022-05-14T11:27:27,026 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:27,829 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:28,526 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:28,527 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:31,343 INFO JControllerProxy-42 c.s.j.c.p.ProxyContext - 'standalone' (https://<host>:<port>): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO JControllerProxy-41 c.s.j.c.p.ProxyContext - 'testsuite' cluster (https://<host>:<port>): ProxyCoupled(1652520420689258)
|
Log Anonymizer Script
Location
Anonymization is available from a Java class and is invoked from the by a shell script available with at the following default locations:
...
| Code Block |
|---|
| title | Invocation of Log Anonymizer Script |
|---|
|
Usage: loganonymize-anonymizerlogs.sh [Options]
Options:
-l | --log-file=<log-file> | optional: location of a log filefiles thatto should be anonymized; filesa single file, directoriesdirectory andor wildcards can be specified;
the argument can occur any number of times
-o | --output-dir=<directory> | optional: output directory for anonymized log files
-r | --rules-file=<rules-file> | optional: the path to a YAML file holding rules for anonymization; by default built-in rules will be applied
-e | --export-rules=<rules-file> | optional: the path to a YAML file to which built-in rules will arebe exported |
Explanation:
- Options
-l | --log-file=<log-file>: Specifies the location of a the log file(s) to be anonymized.- This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified
should - if all included files are to be anonymized.
- Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
- Anonymized log file names are prefixed with the string:
anonymized- - By default anonymized log files are stored in the directory in which original log files are found.
-o | --output-dir=<directory>: Optionally specifies the output directory in which anonymized log files are to be stored. If this argument is omitted then anonymized log files are stored in their original directory.-r | --rules-file=<rules-file>: Optionally specifies the location of a file in YAML format that holds the expressions that should rules to be applied for anonymization.-e | --export-rules=<rules-file>: Optionally specifies the location of a file in YAML format to which default expressions the built-in rules for anonymization are will be exported.
Rules
Rules include to specify specifying regular expressions to search for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.
- Users can export the built-in rules to a file to verify available rules and expressions.
- Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended to validate that individual files to be are validated as being YAML compliant.
| Code Block |
|---|
| language | yml |
|---|
| title | Example for built-in Rules |
|---|
| linenumbers | true |
|---|
|
rules:
- item: url-component
search: ://(.*):(\d{2,5})
replace:
- <host>
- <port>
- item: ip-address
search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))
replace:
- <ip-address>
- item: host-install
search: host[\s]*=[\s]*(.*)
replace:
- <host> |
...
| Code Block |
|---|
| title | Example for Invocation of Log Anonymizer Script |
|---|
|
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/controller/var/logs/controller.log |
Explanation:
- Creates an anonymized version of the log file in the same directory with the name:
anonymized-controller.log
| Code Block |
|---|
| title | Example for Invocation of Log Anonymizer Script for a number of Log Files |
|---|
|
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log \
--log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log \
--output-dir=/tmp/logs |
Explanation:
- Creates anonymized versions of the indicated log files in the given output directory:
/tmp/logs/anonymized-joc.log/tmp/logs/anonymized-joc-debug.log
| Code Block |
|---|
| title | Example for Invocation of Log Anonymizer Script with Wildcards for Log Files |
|---|
|
anonymize-logs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc* \
--output-dir=/tmp/logs |
Explanation:
- Creates anonymized versions of the log files indicated from the given directory. The wildcard specifies any log files carrying a name that starts with
joc will be used. - The wildcard includes consideration of plain text log files with the .log file extension and also consideration of compressed log files with a .gz file extension.
- Anonymized log files are stored in the output directory.
| Code Block |
|---|
| title | Example for Export of Rules to a File |
|---|
|
loganonymize-anonymizerlogs.sh --export-rules=/tmp/rules.yaml
|
Explanation:
- The build-in rules are exported to a file in YAML format.
- This file can be used to adjust rules and can be applied for anonymization later on.
| Code Block |
|---|
| title | Example for Invocation of Log Anonymizer Script with individual Rules from a File |
|---|
|
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/agent/var_4445/logs/*.log \
--output-dir=/tmp/logs \
--rules-file=/tmp/rules.yaml |
Explanation:
- Creates anonymized versions of all log files in the indicated directory and stores them in the output directory.
- A YAML file with individual rules is applied.
Running Log Anonymizer outside of JS7
Users who wish to run the Log Anonymizer outside of a JS7 installation can:
- copy a number of Java related files as available from the Controller or Agent installation,
- create a shell script to invoke the Log Anonymizer.
Java related Files required by Log Anonymizer
The following files can be found in a Controller or Agent installation for Unix or Windows.
- Note that version numbers of binary files can change over time. The examples below assume Java related files as provided with Release 2.4.0.
- Users can copy the following Java related files from the .
/lib/3rd-party, ./lib/sos, ./lib/stdout directories to the location where they want to use Log Anonymizer:
| Code Block |
|---|
| title | Java binary files required by Log Anonymizer |
|---|
| linenumbers | true |
|---|
|
./lib/sos/sos-commons-util-2.4.0.jar
./lib/stdout/log4j2.xml
./lib/3rd-party/org.apache.logging.log4j.log4j-api-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-core-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
./lib/3rd-party/org.slf4j.slf4j-api-1.7.36.jar
./lib/3rd-party/snakeyaml-1.30.jar |
Shell Script to run Log Anonymizer
An individual script to invoke Log Anonymizer can look like this:
| Code Block |
|---|
| title | Example for Invocation of Log Anonymizer from an individual Script on Unix |
|---|
| linenumbers | true |
|---|
|
#!/bin/sh
JAVA_BIN="${JAVA_HOME}/bin/java"
test -x "${JAVA_BIN}" || JAVA_BIN="java"
script_dir="$(echo $(dirname "$0") | cd > /dev/null && pwd)"
cp="-classpath "${script_dir}/lib/sos/*:${script_dir}/lib/3rd-party/*:${script_dir}/lib/stdout""
"${JAVA_BIN}" ${JAVA_OPTIONS} ${cp} com.sos.commons.util.loganonymizer.SOSLogAnonymizer $* |
| Code Block |
|---|
| title | Example for Invocation of Log Anonymizer from an individual Script on Windows |
|---|
| linenumbers | true |
|---|
|
@echo off
set JAVA_BIN=java.exe
if exist "%JAVA_HOME%\bin\java.exe" set JAVA_BIN=%JAVA_HOME%\bin\java.exe
set script_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\stdout"
"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %* |
Explanation:
The directory hierarchy is assumed like this:
sos-commons-util-2.4.0.jar
stdout3rd-party
org.apache.logging.log4j.log4j-api-2.17.2.jarorg.apache.logging.log4j.log4j-core-2.17.2.jarorg.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jarorg.slf4j.slf4j-api-1.7.36.jarsnakeyaml-1.30.jar