...

Hostnames, IP Addresses, Ports
Accounts
Database Names

There are situations when where log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.

...

Anonymization is available from a Java class and is invoked from by a shell script available at the following default locations:

...

Code Block

title	Invocation of Log Anonymizer Script

Usage: loganonymize-anonymizerlogs.sh [Options]

  Options:
    -l | --log-file=<log-file>       | optional: location of a log filefiles that shouldto be anonymized; a single filesfile, directoriesdirectory andor wildcards can be specified;
    -o | --output-dir=<directory>    | optional: output directory of anonymized log files
    -r | --rules-file=<rules-file>   | optional:                       the pathargument tocan aoccur fileany holdingnumber rulesof for anonymizationtimes
    -eo | --exportoutput-rules=<rules-file>dir=<directory>    | optional: theoutput pathdirectory tofor aanonymized filelog tofiles
 which built-in rules are exported

Explanation:

...

-r | --rules-file=

...

<rules-file>

...

This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified if all included files be anonymized.
Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
Anonymized log file names are prefixed with the string: anonymized-
By default anonymized log files are stored in the directory in which original log files are found.

...

Rules

Rules include specifying regular expressions for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.

...

   | optional: path to a YAML file holding rules for anonymization; by default built-in rules will be applied
    -e | --export-rules=<rules-file> | optional: path to a YAML file to which built-in rules will be exported

Explanation:

Options
- -l | --log-file=<log-file>: Specifies the location of the log file(s) to be anonymized.
  - This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified if all included files are to be anonymized.
  - Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
  - Anonymized log file names are prefixed with the string: anonymized-
  - By default anonymized log files are stored in the directory in which original log files are found.
- -o | --output-dir=<directory>: Optionally specifies the output directory in which anonymized log files are to be stored. If this argument is omitted then anonymized log files are stored in their original directory.
- -r | --rules-file=<rules-file>: Optionally specifies the location of a file in YAML format that holds the rules to be applied for anonymization.
- -e | --export-rules=<rules-file>: Optionally specifies the location of a file in YAML format to which the built-in rules for anonymization will be exported.

Rules

Rules include specifying regular expressions for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.

Users can export the built-in rules to a file to verify available rules and expressions.
Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended that individual files are validated as being YAML compliant.

Code Block

language	yml
title	Example for built-in Rules
linenumbers	true

rules:
- item: url-component
  search: ://(.*):(\d{2,5})
  replace:
  - <host>
  -

Code Block

language	yml
title	Example for built-in Rules
linenumbers	true

rules:
- item: url-component
  search: ://(.*):(\d{2,5})
  replace:
  - <host>
  - <port>
- item: ip-address
  search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))
  replace:
  - <ip-address>
- item: host-install
  search: host[\s]*=[\s]*(.*)
  replace:
  - <host>

...

Code Block

title	Example for Invocation of Log Anonymizer Script

loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/controller/var/logs/controller.log

Explanation:

...

Code Block

title	Example for Invocation of Log Anonymizer Script for a number of Log Files

loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log  \
                  --log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log \
                  --output-dir=/tmp/logs

...

Code Block

title	Example for Invocation of Log Anonymizer Script with Wildcards for Log Files

loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc*  \
                  --output-dir=/tmp/logs

...

Creates anonymized versions of the log files indicated from the given directory. The wildcard specifies any log files carrying a name that starts with joc will be used.
The wildcard includes consideration of plain text log files available with the .log file extension and also consideration of compressed log files which are available with a .gz file extension.
Anonymized log files are stored in the output directory.

...

Code Block

title	Example for Export of Rules to a File

loganonymize-anonymizerlogs.sh --export-rules=/tmp/rules.yaml

...

Code Block

title	Example for Invocation of Log Anonymizer Script with individual Rules from a File

loganonymize-anonymizerlogs.sh --log-file=/var/sos-berlin.com/js7/agent/var_4445/logs/*.log \
                  --output-dir=/tmp/logs \
                  --rules-file=/tmp/rules.yaml

Explanation:

Creates anonymized versions of all log files in the indicated directory and stores them in the output directory.
A YAML file with individual rules is applied.

Running Log Anonymizer outside of JS7

Users who wish to run the Log Anonymizer outside of a JS7 installation can

copy a number of binary files as available from the Controller or Agent installation,
create a shell script to invoke Log Anonymizer.

Java Binary Files required by Log Anonymizer

The following binary files can be found from a Controller or Agent installation for Unix or Windows.

...

file=/tmp/rules.yaml

Explanation:

Creates anonymized versions of all log files in the indicated directory and stores them in the output directory.
A YAML file with individual rules is applied.

Running Log Anonymizer outside of JS7

Users who wish to run the Log Anonymizer outside of a JS7 installation can:

copy a number of Java related files as available from the Controller or Agent installation,
create a shell script to invoke the Log Anonymizer.

Java related Files required by Log Anonymizer

The following files can be found in a Controller or Agent installation for Unix or Windows.

Note that version numbers of binary files can change over time. The examples below assume Java related files as provided with Release 2.4.0.
Users can copy the following Java related files from the ./lib/3rd-party, ./lib/sos, ./lib/stdout directories to the location where they want to use Log Anonymizer:

Code Block

title	Java binary files required by Log Anonymizer
linenumbers	true

./lib/sos/sos-commons-util-2.4.0.jar
./lib/stdout/log4j2.xml
./lib/3rd-party/org.apache.logging.log4j.log4j-api-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-core-2.17.2.jar

Code Block

title	Java binary files required by Log Anonymizer
linenumbers	true

./lib/3rd-party/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
./lib/3rd-party/org.slf4j.slf4j-api-1.7.36.jar
./lib/3rd-party/snakeyaml-1.30.jar
./lib/sos/sos-commons-util-2.4.0.jar

Shell Script to run Log Anonymizer

...

Code Block

title	Example for Invocation of Log Anonymizer from an individual Script on Unix
linenumbers	true

#!/bin/sh

JAVA_BIN="${JAVA_HOME}/bin/java"
test -x "${JAVA_BIN}" || JAVA_BIN="java"

script_dir="$(echo $(dirname "$0") | cd > /dev/null && pwd)"
cp="-classpath \"${script_dir}/lib/sos/*:${script_dir}/lib/3rd-party/*:${script_dir}/lib/*:${script_dir}/lib\stdout""

"${JAVA_BIN}" ${JAVA_OPTIONS} ${cp} com.sos.commons.util.loganonymizer.SOSLogAnonymizer $*

Code Block

title	Example for Invocation of Log Anonymizer from an individual Script on Windows
linenumbers	true

@echo off

set JAVA_BIN=java.exe
if exist "%JAVA_HOME%\bin\java.exe" set JAVA_BIN=%JAVA_HOME%\bin\java.exe

set script_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\*;%script_dir%lib"

"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %*

Explanation:

_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\stdout"

"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %*

Explanation:

The directory hierarchy is assumed like this:

anonymize-logs.sh | .cmd
- lib
  - sos

The directory hierarchy is assumed like this:

log-anonymizer.sh | .cmd

lib

3rd-party

- - - sos-commons-util-2.4.0.jar
  - stdout
    - log4j2.xml
  - 3rd-party
    - org.apache.logging.log4j.log4j-api-2.17.2.jar
    - org.apache.logging.log4j.log4j-core-2.17.2.jar
- - - org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
    - org.slf4j.slf4j-api-1.7.36.jar
    - snakeyaml-1.30

.jar

- - - .jar

Space shortcuts

Page tree

Versions Compared

Old Version 19

New Version Current

Key

Rules

Rules

Running Log Anonymizer outside of JS7

Java Binary Files required by Log Anonymizer

Running Log Anonymizer outside of JS7

Java related Files required by Log Anonymizer

Shell Script to run Log Anonymizer

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 19

New Version Current

Key

Rules

Rules

Running Log Anonymizer outside of JS7

Java Binary Files required by Log Anonymizer

Running Log Anonymizer outside of JS7

Java related Files required by Log Anonymizer

Shell Script to run Log Anonymizer