...

• Streaming makes job log output available to JOC Cockpit within 1s. 
• Streaming allows a running log which translates to the fact that users observe ongoing log output of orders and jobs in the log view window.
• In case of missing connections:
  • between Controller and Agent - the Agent stores log output in its journal and forwards logs when reconnected to a Controller later on. 
  • between JOC Cockpit and Controller - the Controller stores log output in its journal and forwards logs when reconnected to JOC Cockpit at a later point in time.

Log Size

Jobs can write log output of arbitrary size to the job's stdout/stderr channel.

• The JS7 Agent and Controller handle log output based on a streaming implementation that can handle any size of log output.
• The JOC Cockpit adds log output to the JS7 database in a compressed format. This processing is affected by large log size by the following aspects:
  • The compression ratio depends on the individual log output, however, it is safe to assume the resulting size of logs in the database to be 10-20% of the original size.
  • The compression process takes more time and more system resources such as CPU and Java Heap Space for larger log output.

The JS7 - History Service that handles logs in JOC Cockpit works with the following threshold settings, see JS7 - Settings:

• A setting for the applicable size of a task log (name: log_applicable_size, default: 500 MB). A task log that exceeds this size will be handled like this:
  • The first 100 KB and the last 100 KB of log output will be used as the task log.
    • A banner information is added to the begin of log output:
      • LOG OUTPUT EXCEEDS APPLICABLE SIZE OF <log_applicable_size> MB AND IS TRUNCATED TO THE FIRST AND LAST 100 KB.
    • The truncated task log will be compressed and added to the database.
  • The original log file will be removed.
  • Occurrence of log output exceeding this threshold raises a warning, however, the task execution history and truncated log output are available from the JOC Cockpit API and GUI.
• A setting for the maximum size of a task log (name: log_maximum_size, default: 1000 MB). A task log exceeding this size will be handled like this:
  • The History Service will stop to write larger log files. Log output exceeding the threshold value is ignored.
  • The first 100 KB of the log file will be used as the task log.
    • A banner information is added to the begin of log output:
      • LOG OUTPUT EXCEEDS MAXIMUM SIZE OF <log_maximum_size> MB AND IS TRUNCATED TO 100 KB.
    • The truncated task log will be compressed and added to the database.
  • The original log file will be removed.

The History Service similarly considers the above settings when performing switch-over and fail-over operations between JOC Cockpit instances (compressing and storing current logs to the database).

The JOC Cockpit GUI makes use of the following threshold setting, see JS7 - Settings:

• A setting for the maximum display size of a task log (name: log_maximum_display_size, default: 10 MB). An uncompressed task log exceeding this size is not offered for display with the Log View window but is offered for download only.

The threshold settings can be modified from the JOC Cockpit Settings page.

Log Files

A number of users prefer to have log files available for each order execution and job execution, for example to consolidate log files on a file server or to submit log files to specific tools for log analysis. Typical shortcomings of log file analysis include:

• Log analysis is useful but usually comes too late. It's more a reporting activity that is not intended to replace near-real time monitoring.
• Log analysis starts when the log is completed. In Unix environments parallel read and write operations on log files can occur. 
• Log analysis is useful when it comes to parsing logs for unexpected output or errors. However, it might prove tedious to teach a log analyzer recovery of errors from log output in case of a job being successfully restarted for a specific order.

...