Introduction

The JS7 offers File Watching out-of-the-box.

Users can implement Directory Watching from jobs.

• Jobs running in JS7 Agents can be used to watch for incoming files from specific changes to directories.
• For each incoming file a file order is created that is passed to the related workflowchange an event is raised to trigger an action.
• Users are in charge of (re)moving files from the incoming watched directories.

File and Directory Watching

There is a fundamental difference between both concepts:

• File Watching is about watching for incoming files from a directory.
• The JS7 will create a file order for each incoming file.
File Watching is performed by JS7 Agents per directory, for details see JS7 - directory, see JS7 - How To - File Watching.
• Directory Watching is about watching for changes to a directory.
  • Changes include the situation when a file or sub-directory is added, changed, renamed or deleted.
  • For each change an event is raised that can be used to trigger actions.
  • Directory Watching is performed by tools and frameworks close to the Operating System such as iNotify, Java and .Net.

Directory Watching

• , see JS7 - How To - Directory Watching

Directory Watching

Find a sample implementation from the following chapters.

The example implements the use case of 

• watching for files that are created in a directory and sub-directories,
• moving incoming files to a target directory.

There is more than one way how to do it, a number of alternative implementations are possibleFind a sample implementation from the following chapters.

Download (upload .json): pdfWatchDirectoryEvents.workflow.json

Workflow

The workflow includes a single job that implements Directory Watching.

...

• The job is included in a JS7 - Cycle Instruction. The purpose of this instruction is to manage periods for which Directory Watching should be performed.
• The workflow makes use of a number of variables.
  • Workflow variables without default value have to be specified when adding orders to the workflow.
    • move_destination: specifies the target directory to which incoming files will be moved.
    • monitor_path: specifies the source directory that is watched for incoming files
  • Workflow variables with default values can be omitted when adding orders to the workflow.
    • monitor_filter: specifies a wildcard expression to select files, for example a filter *.csv will limit files in consideration to this extension.
    • monitor_subdirectories: specifies true if sub-directories should be considered and otherwise false.
    • monitor_log_file: specifies the log file for events.
    • monitor_period: specifies the duration of job execution in seconds. If the specified duration is exceeded then the job will terminate and will be restated by the Cycle Instruction. This guarantees that the file watcher is renewed and that any sub-directories that might have become inaccessible, for example from a remote share that is rebooted, will be registered with the next job run.
    • monitor_interval: specifies the interval in seconds between checks of the monitor_log_file for changes.
    • monitor_steady: specifies the number of seconds that an incoming file is waited for to check that the file size is unchanged and the file can be considered stable.

Cycle

The Cycle Instruction is an option that users apply who want to watch directories during certain business hours only.

...

Clicking the period brings up the details of the cycle:

Job

The job is implemented with PowerShellfor PowerShell 5.1, 6.x, 7.x:

• PowerShell is available for Windows servers by default.
  • Consider the shebang in the first line of the job that indicates the PowerShell version in use:
    • the string @@findstr/v "^@@f.*&" "%~f0"|powershell.exe -&goto:eof indicates use of powershell.exe from version 5.1.
    • the string @@findstr/v "^@@f.*&" "%~f0"|pwsh.exe -&goto:eof indicates use of pwsh.exe from version 6 or 7.
• PowerShell can be installed for Unix environments such as Linux and MacOS.
  
  • Consider to modify the shebang in the first line of the job like this:
    • the string #!/usr/bin/env pwsh can be used assuming that pwsh is the binary of the PowerShell interpreter.
      
      

Find the full source code of the job:

@@findstr/v "^@@f.*&" "%~f0"|powershell.exe -&goto:eof

# for Windows with PowerShell 7 use shebang: @@findstr/v "^@@f.*&" "%~f0"|pwsh.exe -&goto:eof
# for Unix use shebang: #!/usr/bin/env pwsh

# start watcher for a given path, file name filter and optionally any sub-directories
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = $env:MONITOR_PATH
$watcher.Filter = $env:MONITOR_FILTER
$watcher.IncludeSubdirectories = ($env:MONITOR_SUBDIRECTORIES -eq 'true')
$watcher.EnableRaisingEvents = $true   

# define action to move files
$moveFileAction = {
    if ( Test-Path $Event.SourceEventArgs.FullPath -PathType leaf )
    {
    	$lastSize,$size = -1
        While ($true)
        {
            $size = (Get-Item $Event.SourceEventArgs.FullPath).length
            if ( $size -ne $lastSize )
            {
                "$(Get-Date) [Check] $($Event.SourceEventArgs.FullPath) check steady state, size=$size" | Out-File $env:MONITOR_LOG_FILE -Append
				Start-Sleep -Seconds $env:MONITOR_STEADY
                $lastSize = $size
            } else {
                break
            }
        }
        "$(Get-Date) [$($Event.SourceEventArgs.ChangeType)] $($Event.SourceEventArgs.FullPath) moved to $($env:MOVE_DESTINATION)" | Out-File $env:MONITOR_LOG_FILE -Append
        Move-Item -Path $Event.SourceEventArgs.FullPath -Destination $env:MOVE_DESTINATION -Force
    }
}

# define action to log directory events
$logFileAction = {
   "$(Get-Date) [$($Event.SourceEventArgs.ChangeType)] $($Event.SourceEventArgs.FullPath)" | Out-File $env:MONITOR_LOG_FILE -Append
}

# register actions to be taken when events are raised
Register-ObjectEvent -InputObject $watcher -EventName "Created" -Action $moveFileAction | Out-Null
Register-ObjectEvent -InputObject $watcher -EventName "Changed" -Action $logFileAction | Out-Null
Register-ObjectEvent -InputObject $watcher -EventName "Deleted" -Action $logFileAction | Out-Null
Register-ObjectEvent -InputObject $watcher -EventName "Renamed" -Action $logFileAction | Out-Null

# move existing files found before file watcher reports events
Get-ChildItem -Path "$($env:MONITOR_PATH)/$($env:MONITOR_FILTER)" -Recurse | Move-Item -Destination $env:MOVE_DESTINATION -Force

# check existing log file
$sleep,$count,$lastCount = 0
if ( Test-Path $env:MONITOR_LOG_FILE -PathType leaf )
{
	$lastCount = $count = (Get-Content $env:MONITOR_LOG_FILE -ErrorAction silentlycontinue | Measure-Object -Line).Lines
}

# run file watchter for the given period
Write-Output ".. watching for $env:MONITOR_PERIOD seconds"
While( $sleep -le $env:MONITOR_PERIOD )
{
	Start-Sleep -Seconds $env:MONITOR_INTERVAL
    $sleep += $env:MONITOR_INTERVAL
	$count = (Get-Content $env:MONITOR_LOG_FILE -ErrorAction silentlycontinue | Measure-Object -Line).Lines 
    if ( $count -ne $lastCount )
    {
    	Get-Content $env:MONITOR_LOG_FILE -Tail ($count-$lastCount) -ErrorAction silentlycontinue
        $lastCount = $count
    }
}

# perform cleanup
Get-EventSubscriber | Unregister-Event
$watcher.Dispose

Explanation:

• The implemenation makes use of the
  • .Net FileSystemWatcher class
  • PowerShell Register-ObjectEvent cmdlet
• Lines 1 - 1: creates the FileSystemWatcher object
• Lines 1 -1: holds the script block to move files in case of events indicating creation of a file.
• Lines 1 -1: holds the script block to log events in case of events not related to creation of a file.x