Page History
...
- Hostnames, IP Addresses, Ports
- Accounts
- Database Names
There are situations when where log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.
...
Anonymization is available from a Java class and is invoked from by a shell script available at the following default locations:
...
Code Block | ||
---|---|---|
| ||
Usage: loganonymize-anonymizerlogs.sh [Options] Options: -l | --log-file=<log-file> | optional: location of a log filefiles that shouldto be anonymized; a single filesfile, directoriesdirectory andor wildcards can be specified; -o | --output-dir=<directory> | optional: output directory of anonymized log files -r | --rules-file=<rules-file> | optional: the pathargument tocan aoccur fileany holdingnumber rulesof for anonymizationtimes -eo | --exportoutput-rules=<rules-file>dir=<directory> | optional: theoutput pathdirectory tofor aanonymized filelog tofiles which built-in rules are exported |
Explanation:
...
-r | --rules-file= |
...
<rules-file> |
...
- This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified if all included files be anonymized.
- Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
- Anonymized log file names are prefixed with the string:
anonymized-
- By default anonymized log files are stored in the directory in which original log files are found.
...
Rules
Rules include specifying regular expressions for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.
...
| optional: path to a YAML file holding rules for anonymization; by default built-in rules will be applied
-e | --export-rules=<rules-file> | optional: path to a YAML file to which built-in rules will be exported |
Explanation:
- Options
-l | --log-file=<log-file>
: Specifies the location of the log file(s) to be anonymized.- This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified if all included files are to be anonymized.
- Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
- Anonymized log file names are prefixed with the string:
anonymized-
- By default anonymized log files are stored in the directory in which original log files are found.
-o | --output-dir=<directory>
: Optionally specifies the output directory in which anonymized log files are to be stored. If this argument is omitted then anonymized log files are stored in their original directory.-r | --rules-file=<rules-file>
: Optionally specifies the location of a file in YAML format that holds the rules to be applied for anonymization.-e | --export-rules=<rules-file>
: Optionally specifies the location of a file in YAML format to which the built-in rules for anonymization will be exported.
Rules
Rules include specifying regular expressions for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.
- Users can export the built-in rules to a file to verify available rules and expressions.
- Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended that individual files are validated as being YAML compliant.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
rules:
- item: url-component
search: ://(.*):(\d{2,5})
replace:
- <host>
- | ||||||
Code Block | ||||||
| ||||||
rules:
- item: url-component
search: ://(.*):(\d{2,5})
replace:
- <host>
- <port>
- item: ip-address
search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))
replace:
- <ip-address>
- item: host-install
search: host[\s]*=[\s]*(.*)
replace:
- <host> |
...
Code Block | ||
---|---|---|
| ||
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/controller/var/logs/controller.log |
Explanation:
...
Code Block | ||
---|---|---|
| ||
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log \ --log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log \ --output-dir=/tmp/logs |
...
Code Block | ||
---|---|---|
| ||
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc* \ --output-dir=/tmp/logs |
...
- Creates anonymized versions of the log files indicated from the given directory. The wildcard specifies any log files carrying a name that starts with
joc
will be used. - The wildcard includes consideration of plain text log files available with the .log file extension and also consideration of compressed log files which are available with a .gz file extension.
- Anonymized log files are stored in the output directory.
...
Code Block | ||
---|---|---|
| ||
loganonymize-anonymizerlogs.sh --export-rules=/tmp/rules.yaml |
...
Code Block | ||
---|---|---|
| ||
loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/agent/var_4445/logs/*.log \ --output-dir=/tmp/logs \ --rules-file=/tmp/rules.yaml |
...
Users who wish to run the Log Anonymizer outside of a JS7 installation can:
- copy a number of binary Java related files as available from the Controller or Agent installation,
- create a shell script to invoke the Log Anonymizer.
Java
...
related Files required by Log Anonymizer
The following binary files can be found from in a Controller or Agent installation for Unix or Windows.
- Note that version numbers of binary files can change over time. The below examples assume binary files as provided with Release 2.4.0.Users can copy the following binary files from the time. The examples below assume Java related files as provided with Release 2.4.0.
- Users can copy the following Java related files from the .
/lib/3rd-party,
./lib/sos, ./lib/stdout
directories to the location where they want to use Log Anonymizer:
Code Block | ||||
---|---|---|---|---|
| ||||
./lib/sos/sos-commons-util-2.4.0.jar ./lib/stdout/log4j2.xml ./lib/3rd-party/org.apache.logging.log4j.log4j-api-2.17.2.jar ./lib/3rd-party |
...
Code Block | ||||
---|---|---|---|---|
| ||||
/org.apache.logging.log4j.log4j-core-2.17.2.jar ./lib/3rd-party/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar ./lib/3rd-party/org.slf4j.slf4j-api-1.7.36.jar ./lib/3rd-party/snakeyaml-1.30.jar ./lib/sos/sos-commons-util-2.4.0.jar |
Shell Script to run Log Anonymizer
...
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/sh JAVA_BIN="${JAVA_HOME}/bin/java" test -x "${JAVA_BIN}" || JAVA_BIN="java" script_dir="$(echo $(dirname "$0") | cd > /dev/null && pwd)" cp="-classpath "${script_dir}/lib/sos/*:${script_dir}/lib/3rd-party/*:${script_dir}/lib/*:${script_dir}/lib/stdout"" "${JAVA_BIN}" ${JAVA_OPTIONS} ${cp} com.sos.commons.util.loganonymizer.SOSLogAnonymizer $* |
Code Block | ||||
---|---|---|---|---|
| ||||
@echo off
set JAVA_BIN=java.exe
if exist "%JAVA_HOME%\bin\java.exe" set JAVA_BIN=%JAVA_HOME%\bin\java.exe
set script_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\*;%script_dir%lib"
"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %* |
Explanation:
_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\stdout"
"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %* |
Explanation:
The directory hierarchy is assumed like this:
anonymize-logs.sh | .cmd
lib
sos
- The directory hierarchy is assumed like this:
sos-commons-util-2.4.0.jar
stdout
log4j2.xml
3rd-party
org.apache.logging.log4j.log4j-api-2.17.2.jar
org.apache.logging.log4j.log4j-core-2.17.2.jar
org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
org.slf4j.slf4j-api-1.7.36.jar
snakeyaml-1.30
.jar.jar
sos
sos-commons-util-2.4.0
log-anonymizer.sh | .cmd
lib
3rd-party