Page History

...

• User might be interested to automatically forward task logs to some external location.
  • This allows to use log analysis tools such as Splunk® , Elasticsearch®  etc.
  • You are strongly advised not to access task logs from their original location with the JobScheduler Master's ./logs directory for analysis with 3rd party tools.
    • Log files are constantly updated, approx. every 5s, therefore you cannot decide when the log is completed . and you shouldn't block files that JobScheduler requires to access.
    • Log files are stored to the database on task completion and will be overwritten on disk with the next task starting from the same job.
• The forwarding of logs therefore should start from the database and can be scheduled e.g. on a daily basis or more frequently to allow immediate log analysis.

...

The PowerShell CLI can be used by jobs a job to forward logs. This requires prior installation of the JobScheduler PowerShell Module.

The Get-JobSchedulerTaskHistory cmdlet is used to retrieve task history items and to forward them to the Get-JobSchedulerTaskLog cmdlet within a job. Two flavors of the job are available for Windows and Linux. The difference is not about the handling of cmdlets or parameters but due to the fact that PowerShell is invoked differently on Windows and Linux. For Windows environments usually PowerShell is available with the OS, for Linux the job has to call pwsh to invoke the PowerShell.

Please consider that below jobs are examples that have to be adjusted for your environment.

Windows Version for use with Agents

Download: forward_task_logs_windows.job.xml

<?xml version="1.0" encoding="UTF-8" ?>
<job title="Forward Task Logs" process_class="agent_windows">
  <params>
    <param name="history_results_directory" value="/users/jstmp/history"/>
  </params>
  <script language="powershell"><![CDATA[
Import-Module $env:SCHEDULER_DATA/config/powershell/Modules/JobScheduler;
Connect-JS -Url $JOCCockpitUrl -Credential $JOCCockpitCredential | Out-Null;

New-Item -ItemType Directory -Force -Path $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY | Out-Null;

# retrieve last history results if available
if ( Test-Path -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history" -ErrorAction continue )
{
    $lastHistory = Import-Clixml -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history";
} else {
    $lastHistory = Get-JobSchedulerTaskHistory -RelativeDateFrom -8h | Sort-Object -Property startTime;
}
 
# Copy log files to target directory
Get-JSTaskHistory -DateFrom $lastHistory[0].startTime | Tee-Object -Variable lastHistory | Get-JobSchedulerTaskLog | Select-Object @{name='path'; expression={ "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/$(Get-Date $_.startTime -f 'yyyyMMdd-hhmmss')-$([io.path]::GetFileNameWithoutExtension($_.job)).log"}}, @{name='value'; expression={ $_.log }} | Set-Content;
 
# store last history results to a file for later retrieval
$lastHistory | Export-Clixml -Path "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/task.history";
                 
Write-Output ".. logs forwarded to: $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY";
]]></script>
  <run_time/>
</job>

Explanations

• Line 2-3: The job is executed with a Windows Agent that is assigned by a process class.
• Line 3-5: Consider to adjust the history_results_directory parameter to point to a valid directory where to store the log files.
• Line 6: The job is of type "powershell" and will use the Powershell version provided with the server.
• Line 47: The JobScheduler PowerShell module is imported. They The module could be installed with any location in the file system
• Line 78: The Connect-JS cmdlet is used to authenticate with the JOC Cockpit REST Web Service. The required URL and credentials are specified in a PowerShell profile, see PowerShell CLI 1.2 - Use Cases - Credentials Management
• Line 10: The target directory specified by the history_results_directory parameter is created should it not exist.
• Line 13-18: Any previous execution of this job will create a file task.history that stores the latest history entries for which log files have been processed. If such a file exists then the value of the $lastHistory variable is restored and otherwise an initial query about the last 8 hours of task executions is used to populate the $lastHistory variable.
• Line 21: The Get-JobSchedulerTaskHistory cmdlet is called 
  • with the parameter -DateFrom  $lastHistory[0].startTime to specify the most recent history entry to be processed
  Get-JSTaskHistory cmdlet is called 
  • with the parameter -Timezone to specify to which timezone date values in the report should be converted. The parameter value -Timezone (Get-Timezone) specifies that the timezone of the Agent's server is used. Otherwise specify the desired timezone e.g. like this: -Timezone (Get-Timezone -Id 'GMT Standard Time'). Without using this parameter any date values are stored as UTC dates to the report.
    • optionally with additional parameters, e.g. to specify the date range for which
    the report is created 
    • logs are forwarded  A value -DateFrom (Get-Date -Hour 0 -Minute 0 -Second 0).AddDays(-7).ToUniversalTime() specifies that
    the report should cover
    • logs should be forwarded for the last 7 days (from midnight). Keep in mind that dates have to be specified for the UTC
    timezone
    • time zone. Without this parameter
    the report
    • logs will be
    created
    • forwarded for the last day.
    • see the Get-
    JSTaskHistory
    • JobSchedulerTaskHistory cmdlet for a full parameter reference.
  Line 11-19: From
  • the output of this cmdlet is pipelined to the
  Get
  • Tee-Object cmdlet to store new values to the $lastHistory variable and to proceed with the pipeline.
  • the following pipeline step includes to read the task log for each entry reported by the Get-JobSchedulerTaskHistory cmdlet.
  • the next pipeline step includes to select properties from the pipeline that are forwarded to the Set-Content cmdlet that expects the Path and Value parameters. For the Path parameter the output directory and the file name for the log file are specified from the task's start time and job name.
• Line 24: Finally the values returned by the $lastHistory variable are persistently written to disk for later retrieval.JSTaskHistory cmdlet a number of properties are selected and and are specified for the sequence in which they should occur in the report. 
  • To add more speaking column headers the property names are mapped to a more readable textual representation.
  • Consider the handling of date formats in lines 15, 16. Use of the Get-Date cmdlet converts the output format of dates (not the timezone) to the default format that is in place on the Agent's server. Without using the Get-Date cmdlet any date values will be stored to the report in ISO format, e.g. 2020-12-31 10:11:12+02:00 for a date in the European central timezone that is UTC+1 in winter time and UTC+2 in summer time.
  • Lines 17 introduces a new property, a calculated duration. From the start time and end time values of a past start the difference in seconds is calculated and is forwarded to the report.
  Line 20: The list of properties per task history item is piped to the Export-Excel cmdlet that is available with the ImportExcel PowerShell Module. The report file name is specified and optionally the worksheet. For a full list of parameters see the ImportExcel PowerShell Module.

Linux Version

Download: reportforward_task_historylogs_linux.job.xml

<?xml version="1.0" encoding="UTF-8" ?>
<job title="ReportForward Task HistoryLogs" process_class="agent_linux">
  <params>
    <param name="history_results_directory" value="/tmp/history"/>
  </params>
  <script language="shell"><![CDATA[
pwsh -NoLogo -NonInteractive -Command '& {
    . $env:SCHEDULER_DATA/config/powershell/JobScheduler.PowerShell_profile.ps1;
    Import-Module $env:SCHEDULER_DATA/config/powershell/Modules/ImportExcel;
    Import-Module $env:SCHEDULER_DATA/config/powershell/Modules/JobScheduler;

    Connect-JS -Url $JOCCockpitUrl -Credential $JOCCockpitCredential | Out-Null;
 
    # Dates in local timezone, output includes local date format
    Get-JSTaskHistory -Timezone (Get-Timezone ) `
 New-Item -ItemType Directory -Force -Path $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY | Out-Null;
 
    # retrieve last history results if available
    if |(  SelectTest-ObjectPath -Property @{name="JobScheduler ID"; expression={$_.jobschedulerId}}, `
  Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history" -ErrorAction continue )
    {
        $lastHistory =                            @{name="Task ID"; expression={$_.taskId}}, `Import-Clixml -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history";
   } else {
        $lastHistory = Get-JobSchedulerTaskHistory -RelativeDateFrom -8h | Sort-Object                        @{name="Job"; expression={$_.job}}, `
                                           @{name="Status"; expression={$_.state._text}}, `-Property startTime;
    }
 
    # Copy log files to target directory
                                 @{name="Start Time"; expression={ Get-Date $_.startTime }}, `
                                           @{name="End TimeGet-JSTaskHistory -DateFrom $lastHistory[0].startTime | Tee-Object -Variable lastHistory | Get-JobSchedulerTaskLog | Select-Object @{name="path"; expression={ "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/$(Get-Date $_.endTime }}, `
                                           @{name="Duration (sec.)"; expression={ (New-Timespan -Start "$($_.startTime)" -End "$($_.endTime)").Seconds }}, `
                                           startTime -f 'yyyyMMdd-hhmmss')-$([io.path]::GetFileNameWithoutExtension($_.job)).log"}}, @{name="Criticalityvalue"; expression={ $_.criticalitylog }}, ` | Set-Content;
 
    # store last history results to a file for later retrieval
    $lastHistory | Export-Clixml                      @{name="Exit Code"; expression={$_.exitCode}} `-Path "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/task.history";
                | Export-Excel -Path /tmp/jobscheduler_reporting.xlsx -WorksheetName "Task-History" -ClearSheet;

    Write-Output ".. logs reportforwarded createdto: /tmp/jobscheduler_reporting.xls$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY";
}'
]]></script>
  <run_time/>
</job>
</job>

...

• Basically the same explanations as for the Windows version of the job applyapplies.
• Line 47: The PowerShell has to be invoked with pwsh. Consider that any subsequent PowerShell commands are quoted within a string that starts with line 3 7 and that ends with line 29. 
  • As the string is using a single quote all subsequent PowerShell commands make use of double quotes when required.
  • You could apply a different quoting style, however, quotes have to be consistent.
• Line 58: As an example a PowerShell profile is invoked that provides the variables for URL and credentials to access the JOC Cockpit REST Web Service. Such profiles can be stored in different locations and can be invoked automatically by pwsh on startup.

...