...
Our recommended approach for this situation would be:
| Graphviz |
|---|
digraph "Example: File Transfer using JadeJob"\{
graph [rankdir=TB]
node [shape=Mrecord,style="filled",fillcolor=lightblue]
node [fontsize=10]
ranksep=0.3
subgraph cluster_0 \{
style=filled;
color=lightgrey;
node [style=filled,color=white];
label = "JobScheduler";
labeljust = "r";
pad = 0.1;
FileOrderSource[label= "\{<f0> FileOrder\nSource|<f1>Identify\ Category|<f2>Is\ Lock\ Set|\{<f3>Yes|<f4>No\}\}"];
StartProcessing[label= "\{<f0>Start\ Processing\ File\}"];
EndProcessing[label= "\{<f0>End\ Processing\ File|<f1>Release\ Lock\}"];
FileOrderSource:f3 -> Setback -> FileOrderSource:f2;
FileOrderSource:f4 -> StartProcessing -> EndProcessing;
\}
File_Transfer -> FileOrderSource;
\} |
- Use a single 'load_file' File Order Source directory to which all files are delivered.
- JobScheduler would use regular expressions to identify the files arriving in this directory on the basis of their names, timestamps or file extensions and forward them for processing accordingly.
- JobScheduler would then set a 'lock' for the subsidiary whose file is being processed to prevent further files from this subsidiary being processed as long as processing continues.
No Format Should a 'new' file from this subsidiary arrive whilst its predecessor is being processed, the job 'receiving' the new file will be 'set back' by JobScheduler as long as the lock for the subsidiary is set.
- This lock would be released by JobScheduler once processing of the 'first' file has been completed.
- The job 'receiving' the new file will now be able to forward the new file for processing.
...
- JS starts as soon as file matching with Regular Expression found in the directory.
No Format This directory is set in the "File Order Sources" area in the "Steps/Nodes" view of the "load_files" job chain. TODO: SCREENSHOT
- JS's aquire_lock job matches file with regular expression and decide file's category: for example Berlin or Munich
No Format The regular expressions are also defined in the "File Order Sources" area shown in the screenshot. Note that the expressions match with the prefixes "a" and "b" in the file names and not directly "Berlin" or "Munich". - Once aquire_lock finds the matching category it will try to set an Semaphore (Flag) using JS's inbuilt LOCK mechanism
- There is only one instance on LOCK is allowed and once the LOCK has been assigned to first file of Berlin category, next file has to wait with a setback until the LOCK is free.
- THe same mechanism will be repeated for files from category Munich but since the LOCK is not acquired ( or Semaphore (Flag)) is not set for Munich, file from category Munich will be allowed to be processed.
- Once process is finished depending upon success or error , JS will move the file from in to either done (success) or failed(error).
- After moving input file to correct target directory JS job release_lock will be called which will remove the lock/Semaphore from JS and next file from same category will be allowed.
...