Page History

...

The Background Service can collect file transfer records from any number of JADE Clients and store this information in a database. The History Viewer web interface is available for examining ongoing file transfer records and provides comprehensive filtering for sorting records. In addition notifications can be sent to operators and monitoring systems.

...

• This tutorial uses the simple FTP file transfer example described in the The JADE YADE Client Command Line Interface - Tutorial 1 - Getting Started article as a basis for demonstrating the configuration of the Background Service.
• In addition, the use of the XML Editor to generate the settings.ini file used to specify the file transfer and Background Service configurations is described. This editor generates both the settings.ini configuration file required for JADE versions up to and including 1.10 and the XML configuration file required for version 1.11 onwards as well as validating the input against an XSD schema before generating the configuration files. This will considerably simplify the configuration process and reduce development and testing requirements. 
• Links behind parameter names are to the parameter reference article for that parameter.

Other JADE tutorials

See the JADE YADE - Tutorials article for an overview of other JADE tutorials.

...

• Downloads:
  • JobScheduler (Background Service Automation Jobs are included in the JobScheduler download)
  • JADE Client and Background Service History Viewer (available as separate downloads)
  • XML Editor
• Installation Instructions:
  
  • JobScheduler
  • JADE and Background Service History Viewer
  • XML Editor

Background Service Information Retrieval Modes

The JADE Background Service Automation Jobs import file transfer records generated after each file transfer operation has been completed in two independent but complimentary modes:

• Immediate mode:
  • After completion of a file transfer operation the JADE Client sends detailed history records using the UDP or TCP protocols to a port that is being monitored by the JobScheduler providing the Background Service.
    
    • The UDP protocol operates in a "fire and forget" manner - i.e. without feedback - so that the JADE Client is not aware whether the history file has been received by the Background Service.
    • Note that we strongly recommend that the UDP protocol is used here and not TCP as this has the advantage that errors will not occur in the event of a network connection not being able to be made.
• Batch mode:
  • This is provided by the JADE YADE Background Service Automation Jobs - a job chain run by the JobScheduler providing the Background Service. This job chain reads a persistently stored CSV history file in which the JADE Client writes file transfer records after each transfer operation has been completed. The Background Service Automation job chain then saves this information to the Background Service database.

...

We have designed the two Background Service modes to be used together to provide quick and reliable information about the operations that have been completed by JADE:

• Immediate mode transfer per UDP/TCP is almost instantaneous but in the event of, for example, a network disruption, the records being transferred will be lost.
  • The records being transfer are received by a job running on the JobScheduler providing the Background Service and written to the Background Service database.
• The Background Service Automation batch mode is intended to be run at regular intervals by the JobScheduler providing the Background Service. This mode effectively provides a backup for the UDP/TCP transfer mode.
  • The Background Service Automation jobs read file transfer records that the JADE Client has saved in its persistent history file and writes these records to the Background Service database.

...

• the ftp_server_2_local demonstration FTP fragment:
  • This fragment configuration has already been used in other tutorials in this series and downloads a number of files from an online FTP server.
• the BackgroundServiceFragment Fragment:
  
  • This fragment specifies the JobScheduler service that the JADE Client is to send the file history information to immediately after completion of each file transfer operation.
    
    • The configuration of the four child elements required for the BackgroundServiceFragment can be taken from the JADE YADE - Installation - Background Service article.
      
      

3) Configuration of the Profile Elements

The file transfer profile and Background Service elements are is configured next.

• Profile Configuration

  • The Profile element is almost identical to the simple ftp profile configuration used for in other tutorials in this series except that in this tutorial the target destination is a sub-folder of the jade_demo folder.
    The Profile configuration is shown in the screen shot below.
    
    

  

4) Configuration of the General History and Background Service Elements

The transfer Transfer History and Background Service elements are configured next.

...

• The Background Service Element

  • The Background Service fragment configured in the previous section is called using the BackgroundServiceFragmentRef element shown in the screenshot below:
    
    
    
    

• The History Element

  • The generation of the persistent jade_history.csv file for the Background Service Automation is configured using the two parameters shown in the screenshot above: HistoryFile and HistoryFileAppendMode.
     

  • HistoryFile
    It is assumed that the JADE Client(s) and Background Service JobScheduler are installed on separate servers. For this situation the Background Service has a job chain which transfers files that individual JADE Clients have written on their local file system to the directory on the Background Service file system that is monitored by the jade_history_file_order job chain.
    • The Background Service monitors by default the directory:
      • 
        
        
  • HistoryFileAppendMode
    Setting this parameter to true means that a number of file transfer operations can be written to this file before it is processed by the Background Service. This has the advantage that it allows file transfer records to be 'collected' in the file and the reading of the file to be set to a time for less frequent file access.

...

It has already been mentioned above that the JADE Background Service parameters are effectively global settings that are called each time a file transfer operation is started from a configuration file where the Background Service is specified.

As shown in the The JADE YADE Client Command Line Interface - Tutorial 1 - Getting Started article, the ftp_server_2_local demonstration operation is started by entering the following in the command line:

...

• jade_history
  • this job chain is started each time a file transfer record is sent by the JADE Client by either UDP or TCP
  • the job chain writes the file transfer record to the Background Service database
• jade_history_receive
  • this job is used to collect history files from a number of servers that the JADE Client is operated for. This job makes use of SFTP to transfer the history files to the JobScheduler server. The history files are then processed by the jade_history_file_order.
  • this is an order job and a sample order can be found in the following JobScheduler file:
    • $SCHEDULER_DATA\config\live\sos\jade\jade_history_receive,receive.order.xml_sample
  • This order can be activated by by removing the _sample suffix from the file name.
  • The order can then be configured by opening the $SCHEDULER_DATA\config\live\sos\jade directory in JOE (the JobScheduler Object Editor), where it will appear against the ID/Title receive - receive as shown in the next screenshot.
• jade_history_file_order
  • this job chain monitors a directory and by default is configured to process all history files landing in this directory.
    The default location of this directory is:
    
    • $SCHEDULER_DATA\jade_history\
  
  
  
  
  
  • Note that the order comes with a full set of sample parameters.
    
    • The screenshot shows the parameters after they have been modified to suit the Background Service configuration specified in this tutorial, where the history record file has been generated into a local directory and the transfer takes place from the source_dir shown in the screenshot to the ./jade_history target directory that is monitored by the jade_history_file_order job.
    • The source_user and source_password parameters have been deleted as they are not required for a local to local file transfer.
  • The sample order comes with a Run Time configuration for certain days of the month as can be seen in the screenshot.
    • For the purpose of this tutorial these parameters can either be modified so that the job chain will run in a few minutes or the order can be started on demand from the JobScheduler's JOC interface (JobScheduler Operating Interface) as described in the JobScheduler - Tutorial 1 - Running a simple job using JOC article.
  • This order will copy the history file generated by JADE to the target folder where it will be briefly visible until it is processed by the jade_history_file_order job chain before being deleted.
  • An instance of this order has to be started for each JADE client in the Background Service.

History Viewer

The following screenshot shows the History Viewer output for the tutorial file transfer.

...