Introduction
Log files include data that are considered sensitive by a number of users. This includes
- Hostnames, IP Addresses, Ports
- Accounts
- Database Names
There are situations when log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.
JS7 offers to anonymize logs by replacing sensitive data with placeholders:
2022-05-14T11:27:27,026 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-seconda 2022-05-14T11:27:27,829 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443) 2022-05-14T11:27:28,526 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'standalone' (https://controller-2-0-standalone:4443) 2022-05-14T11:27:28,527 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'standalone' (https://controller-2-0-standalone:4443) 2022-05-14T11:27:31,343 INFO JControllerProxy-42 c.s.j.c.p.ProxyContext - 'standalone' (https://controller-2-0-standalone:4443): ProxyCoupled(1652478862797000) 2022-05-14T11:27:32,908 INFO JControllerProxy-41 c.s.j.c.p.ProxyContext - 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443): ProxyCoupled(1652520420689258)
2022-05-14T11:27:27,026 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'testsuite' cluster (https://<host>:<port>) 2022-05-14T11:27:27,829 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'testsuite' cluster (https://<host>:<port>) 2022-05-14T11:27:28,526 INFO main c.s.j.c.p.ControllerApiContext - connect ControllerApi of 'standalone' (https://<host>:<port>) 2022-05-14T11:27:28,527 INFO main c.s.j.c.p.ProxyContext - start Proxy of 'standalone' (https://<host>:<port>) 2022-05-14T11:27:31,343 INFO JControllerProxy-42 c.s.j.c.p.ProxyContext - 'standalone' (https://<host>:<port>): ProxyCoupled(1652478862797000) 2022-05-14T11:27:32,908 INFO JControllerProxy-41 c.s.j.c.p.ProxyContext - 'testsuite' cluster (https://<host>:<port>): ProxyCoupled(1652520420689258)
Log Anonymizer Script
Location
Anonymization is available from a Java class and is invoked from the a shell script available with the following default locations:
# JOC Cockpit /opt/sos-berlin.com/js7/joc/jetty/bin/anonymize-logs.sh # Controller /opt/sos-berlin.com/js7/controller/bin/anonymize-logs.sh # Agent /opt/sos-berlin.com/js7/agent/bin/anonymize-logs.sh
@rem JOC Cockpit C:\Program Files\sos-berlin.com\js7\joc\jetty\bin\anonymize-logs.cmd @rem Controller C:\Program Files\sos-berlin.com\js7\controller\bin\anonymize-logs.cmd @rem Agent C:\Program Files\sos-berlin.com\js7\agent\bin\anonymize-logs.cmd
Usage
Usage: log-anonymizer.sh [Options] Options: -l | --log-file=<log-file> | optional: location of a log file that should be anonymized; files, directories and wildcards can be specified -r | --rules-file=<rules-file> | optional: the path to a file holding rules for anonymization -e | --export-rules=<rules-file> | optional: the path to a file to which built-in rules are exported
Explanation:
- Options
-l | --log-file=<log-file>
: Specifies the location of a log file to be anonymized. This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified should all included files be anonymized.-r | --rules-file=<rules-file>
: Optionally specifies the location of a file in YAML format that holds the expressions that should be applied for anonymization.-e | --export-rules=<rules-file>
: Optionally specifies the location of a file to which default expressions for anonymization are exported.
Rules
Rules include to specify regular expressions to search for and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc.
- Users can export the built-in rules to a file to verify available rules and expressions.
- Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended to validate individual files to be YAML compliant.
rules: - item: url-component search: ://(.*):(\d{2,5}) replace: - <host> - <port> - item: ip-address search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])) replace: - <ip-address> - item: host-install search: host[\s]*=[\s]*(.*) replace: - <host>
Explanation:
- Each
item
specifies an expression tosearch
for and toreplace
. - The
search
expression makes use of capturing groups specified by( ... )
. - The
replace
placeholder specifies a number of strings that replace the content of related capturing groups.
Examples
The following examples illustrate typical use cases.
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/controller/var/logs/controller.log
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log \ --log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log
log-anonymizer.sh --export-rules=/tmp/rules.yaml
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/agent/var_4445/logs/*.log \ --rules-file=/tmp/rules.yaml