JS7 JobScheduler

Space shortcuts

Page tree

Introduction

Users can implement Directory Watching from jobs.

  • Jobs running in JS7 Agents can be used to watch for changes to directories.
  • For each change an event is raised to trigger an action.
  • Users are in charge of (re)moving files from the watched directories.

File and Directory Watching

There is a fundamental difference between both concepts:

Configuring Directory Watching

Find a sample implementation from the following chapters.

The example implements the use case of 

  • watching for files that are created in a directory and sub-directories,
  • moving incoming files to a target directory.

There is more than one way how to do it, a number of alternative implementations are possible.

Download

Download (upload .json): pdfWatchDirectoryEvents.workflow.json

Workflow

The workflow includes a single job that implements Directory Watching.


Explanation:

  • The job is included in a JS7 - Cycle Instruction. The purpose of this instruction is to manage periods for which Directory Watching should be performed.
  • The workflow makes use of a number of variables.
    • Workflow variables without default value have to be specified when adding orders to the workflow.
      • move_destination: specifies the target directory to which incoming files will be moved.
      • monitor_path: specifies the source directory that is watched for incoming files
    • Workflow variables with default values can be omitted when adding orders to the workflow.
      • monitor_filter: specifies a wildcard expression to select files, for example a filter *.csv will limit files in consideration to this extension.
      • monitor_subdirectories: specifies true if sub-directories should be considered and otherwise false.
      • monitor_log_file: specifies the log file for events.
      • monitor_period: specifies the duration of job execution in seconds. If the specified duration is exceeded then the job will terminate and will be restated by the Cycle Instruction. This guarantees that the file watcher is renewed and that any sub-directories that might have become inaccessible, for example from a remote share that is rebooted, will be registered with the next job run.
      • monitor_interval: specifies the interval in seconds between checks of the monitor_log_file for changes.
      • monitor_steady: specifies the number of seconds that an incoming file is waited for to check that the file size is unchanged and the file can be considered stable.

Cycle

The Cycle Instruction is an option that users apply who want to watch directories during certain business hours only.

The instruction allows specifying weekdays, month days etc. and the period during the day for which the job will be repeatedly executed.

Clicking the Cycle Instruction in the Configuration view displays the cycle in the right panel from the property editor.


Clicking the period brings up the details of the cycle:

Job

The job is implemented for PowerShell 5.1, 6.x, 7.x:

  • PowerShell is available for Windows servers by default.
    • Consider the shebang in the first line of the job that indicates the PowerShell version in use:
      • the following string indicates use of powershell.exe from version 5.1:
        • @@setlocal enabledelayedexpansion & @@findstr/v "^@@[fs].*&" "%~f0" | powershell.exe -NonInteractive -Command - & exit !errorlevel!/b&
      • the following string indicates use of pwsh.exe from version 6 or 7:
        • @@setlocal enabledelayedexpansion & @@findstr/v "^@@[fs].*&" "%~f0" | powershell.exe -NonInteractive -Command - & exit !errorlevel!/b&
  • PowerShell can be installed for Unix environments such as Linux and MacOS.
    • Consider modifying the shebang in the first line of the job like this:
      • the string #!/usr/bin/env pwsh can be used assuming that pwsh is the binary of the PowerShell interpreter.

Running the Workflow

Adding an Order

In the Workflows view users can add an order to the workflow:

  • The workflow's action menu offers the Add Order operation.
  • Alternatively the icon can be used to add an order.


Adding an order brings up the following popup window:

  • The popup window displays the list of required workflow variables. An order can be added only after values for required variables are specified.
  • Workflow variables equipped with a default value can be changed when clicking the Modify Variables link.


When clicking the Modify Variables link the following popup window is displayed:

  • The popup window allows to specify values for all variables declared with the workflow.



Monitoring Execution

When the order is added it will be displayed in the Workflows view

  • from a green bullet in the same row as the job that is executed.
  • from the History panel the Order ID is visible that can be clicked to open the order's log view window.

Viewing the Order Log

The order log view window displays

  • information about the start time and the Agent used to execute the workflow,
  • information about workflow variables,
  • any output created by the job,
  • the contents of the monitor log that holds registered events.

Adding a Schedule

If the workflow runs fine you can automate execution by adding a schedule, see JS7 - Schedules.

  • Assign the newly created workflow to your schedule.
  • The list of workflow variables will be displayed similarly to adding an ad hoc order.


Using the Run-time button brings up the popup window to specify start time rules:

  • Adding a schedule requires to assign a working day calendar, see JS7 - Calendars and Schedules.
  • For the start time add a period with a single start at midnight of your time zone.
    • The workflow's Cycle Instruction will rule the point in time during the day when the order will start on the given days.
  • Optionally add a Restriction to limit execution of the workflow to certain days, for example Mon-Fri.

Backup

Job Source Code

Find the full source code of the job:

Job script source 
@@findstr/v "^@@f.*&" "%~f0"|powershell.exe -&goto:eof

# for Windows with PowerShell 7 use shebang: @@findstr/v "^@@f.*&" "%~f0"|pwsh.exe -&goto:eof
# for Unix use shebang: #!/usr/bin/env pwsh

# start watcher for a given path, file name filter and optionally any sub-directories
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = $env:MONITOR_PATH
$watcher.Filter = $env:MONITOR_FILTER
$watcher.IncludeSubdirectories = ($env:MONITOR_SUBDIRECTORIES -eq 'true')
$watcher.EnableRaisingEvents = $true   

# define action to move files
$moveFileAction = {
    if ( Test-Path $Event.SourceEventArgs.FullPath -PathType leaf )
    {
    	$lastSize,$size = -1
        While ($true)
        {
            $size = (Get-Item $Event.SourceEventArgs.FullPath).length
            if ( $size -ne $lastSize )
            {
                "$(Get-Date) [Check] $($Event.SourceEventArgs.FullPath) check steady state, size=$size" | Out-File $env:MONITOR_LOG_FILE -Append
				Start-Sleep -Seconds $env:MONITOR_STEADY
                $lastSize = $size
            } else {
                break
            }
        }
        "$(Get-Date) [$($Event.SourceEventArgs.ChangeType)] $($Event.SourceEventArgs.FullPath) moved to $($env:MOVE_DESTINATION)" | Out-File $env:MONITOR_LOG_FILE -Append
        Move-Item -Path $Event.SourceEventArgs.FullPath -Destination $env:MOVE_DESTINATION -Force
    }
}

# define action to log directory events
$logFileAction = {
   "$(Get-Date) [$($Event.SourceEventArgs.ChangeType)] $($Event.SourceEventArgs.FullPath)" | Out-File $env:MONITOR_LOG_FILE -Append
}

# register actions to be taken when events are raised
Register-ObjectEvent -InputObject $watcher -EventName "Created" -Action $moveFileAction | Out-Null
Register-ObjectEvent -InputObject $watcher -EventName "Changed" -Action $logFileAction | Out-Null
Register-ObjectEvent -InputObject $watcher -EventName "Deleted" -Action $logFileAction | Out-Null
Register-ObjectEvent -InputObject $watcher -EventName "Renamed" -Action $logFileAction | Out-Null

# move existing files found before file watcher reports events
Get-ChildItem -Path "$($env:MONITOR_PATH)/$($env:MONITOR_FILTER)" -Recurse | Move-Item -Destination $env:MOVE_DESTINATION -Force

# check existing log file
$sleep,$count,$lastCount = 0
if ( Test-Path $env:MONITOR_LOG_FILE -PathType leaf )
{
	$lastCount = $count = (Get-Content $env:MONITOR_LOG_FILE -ErrorAction silentlycontinue | Measure-Object -Line).Lines
}

# run file watchter for the given period
Write-Output ".. watching for $env:MONITOR_PERIOD seconds"
While( $sleep -le $env:MONITOR_PERIOD )
{
	Start-Sleep -Seconds $env:MONITOR_INTERVAL
    $sleep += $env:MONITOR_INTERVAL
	$count = (Get-Content $env:MONITOR_LOG_FILE -ErrorAction silentlycontinue | Measure-Object -Line).Lines 
    if ( $count -ne $lastCount )
    {
    	Get-Content $env:MONITOR_LOG_FILE -Tail ($count-$lastCount) -ErrorAction silentlycontinue
        $lastCount = $count
    }
}

# perform cleanup
Get-EventSubscriber | Unregister-Event
$watcher.Dispose


Explanation:

  • The implemenation makes use of the
  • Lines 7 - 11: creates the FileSystemWatcher object
  • Lines 14 - 33: hold the script block to move files in case of events indicating creation of a file.
  • Lines 36 -38: hold the script block to log events in case of events not related to creation of a file.
  • Lines 41 - 44: subscribe to the related events and assign the respective script block
  • Lines 58 - 68: loops for the given period and interrupts in intervals to check for changes to the monitor log file.
  • Lines 71 - 72: unsubscribes from registered events and destroys the FileSystemWatcher object.