JS7 JobScheduler

Space shortcuts

Page tree

Versions Compared

Old Version 8

changes.mady.by.user Andreas Püschel

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Püschel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Introduction

  • JS7 is a rewrite from scratch of the JobScheduler componentsproducts from scratch. The motivation for the developing JS7 is was not to improve the existing branch 1.x x branch (JS1) but to create something better.
  •  The code base of JS1 partly dates back some 12 years, major parts had been written in C++ for an object oriented design. This makes There are three reasons to rewrite for rewriting JS7:
    • A code base needs a redesign from time to time, : you cannot indefinitely continue to add bells and whistles : when. for example, if the code was not designed for to meet more elaborate security requirements that which were not known or evident at the time of writing. The JS7 is designed to meet up-to-date requirements for performance, resilience and security.
    • The C++ code base requires means that a number of software packages that have to be installed in the operating system. In addition, it does not allow perfect platform independence. The JS7 makes use of Java for the purpose of guaranteed platform independence and minimum installation requirements.
    • The object oriented design of JS1 is not a perfect match for ongoing development of the product. You can design an object oriented model with future development of a product in mind for a foreseeable time only, maybe for some two to four years, but you cannot design a model that is stable for a product life cycle of eight years. For the JS7 Controller and Agent SOS switched to the Scala programming language using a functional design for most parts of the JS7 Controller and Agent. This is due to the fact that functional programming is free from side effects, whereas an object oriented approach tends to add more and more relationships to objects that makes ongoing product development increasingly complex.
  • The first release of JS7 is focused on new users of the product. Existing users of the JobScheduler will find migration tools that are being improved in the course of JS7 releases, See  See the JS7 - Migration of JobScheduler 1.x job configurationJob Configuration article for more information.

Architecture Changes

Database

Run-time Database dropped

  • The JS1 Master requires use of a run-time database by the Master to store job results and logs immediately after execution of a job.
  • This database is has been dropped as the JS7 - History Service performs this task asynchronously.

Inventory/Reporting Database

  • The JS7 and JS1 Reporting Database exists in a similar way for JS7.both require a Reporting Database for similar purposes. However, this is not the same database: initial installation of JS7 has to start from a new database or schema, see JS7 - Database.
    • Users who want to access the job history and logs of JS1 are recommended to continue use of JS1 JOC Cockpit and its database with the JS1 Master being deactivated.
  • The However, the JS7 - Database stores inventory information and deployment information for job-information about inventory, deployment and execution history of jobs, workflows and related configuration. This
    • Inventory and deployment: The database is not required at run-time but at design-time and at deploy-time only.
    Therefore
    • In this respect high availability requirements
    to this database are considered being more relaxed by a number of users
    • for the database can be considered to be more relaxed.
    • Execution history: If the database is not available then the execution history and logs cannot be made available in near-real time with JOC Cockpit. This does not affect execution of jobs and workflows. The execution history is not lost but will be streamed from the Controller to JOC Cockpit at a later point in time when the database becomes available.
    • API Service: The database is required for use of the JS7 - REST Web Service API. Users who run jobs with access to the API might prefer to use a highly available database.

Controller / Agent

Agent Assignment

  • In JS1 an Agent can be accessed by a number of Master instances.
  • With JS7 Agents are dedicated to a Controller and cannot be are not shared between Controllers.

Agent Cluster Configuration

  • For JS1 any number of Agent Clusters can be configured that reference the same Agent as they represent a logical layer on top of the installation.
  • With JS7 there is a unique implements a new JS7 - Agent Cluster with unique registration of an Agent Agents during JS7 - Initial Operation.

Configuration Changes

Job-related Configuration Format

XML

...

Format migrated to JSON

  • The JS1 stores job-related configuration in XML files. The problem with XML is that many parsers are inherently insecure due to the many capabilities of XML. One more problem is the fact that a number of XML parser projects are not actively maintained. This results in an increasing number of XML vulnerabilities and inacceptable delays for with security fixes.
  • The JS7 makes use of JSON as the storage format for uses JSON as the JS7 - Inventory Storage Format for job-related configuration. In fact Technically JSON capabilities are by far inferior to what is available from XML , e.g. when it comes to validation. At the same time JSON is more focused and offers less attack surface.

File based Configuration migrated to

...

Database

  • The file based configuration of JS1 is not perfectly cloud usable. This is why branch 1.13 dropped the JOE job editor and introduced browser based management of job-related configuration configurations stored with the database. However, JS1 allows modifications of job-related configuration by files for users with access to the operating system. From a security perspective this is not a perfect design as technical permissions of which allow a system administrator to modify files at operating system level are not related to role based permissions that determine who should be entitled to modify and to deploy jobs.
  • The JS7 follows a role based access model that excludes OS users with technical permissions such as a root account with technical permissions for file systems. Instead, digital signatures are used to prove to a Controller and Agent that a specific user is entitled to deploy certain workflows and jobs - see JS7 - Secure Deployment of Scheduling Objects for more information. This adds non-repudiability to job-related deployment that is not based on trust relationships of for JS7 components products but purely on the use of certificates only, e.g. . For example, an Agent does not trust a Controller or JOC Cockpit, it only trusts the available certificates only.
  • For lovers of the straightforward option available from with JS1 to push/pull job-related configuration files to/from a version control system, the good news is: the JS7 includes direct support for Git based repositories, see the JS7 - RoadmapGit Repository Interface. Support for additional version control products becomes is available from file based exports of the JS7 inventorythe JS7 - Inventory Export and Import.

Job Dependencies

Job

...

Dependencies migrated to

...

Workflow Patterns

Individual Implementation of

...

Dependencies moved to

...

Workflow Patterns

  • Due to the limited capabilities of the JS1 for backward dependencies users might may have implemented dependencies individually, e.g. at an individual level - for example, by scripting. the The Product Knowledge Base includes a number of such examples.
  • The JS7 implements a larger number of workflow patterns as stated with in the previous chaptersection. SOS does not consider it desirable to bury the logic of job dependencies in scripts. Instead such scripts should be migrated to use JS7 - Workflow Instructions and should be part of the configuration. This is manual work and is not supported by migration tools.

Pre-Post Processing

Support for Scripting Languages migrated to GraalVM

  • JS1 includes support for a number of scripting languages:
    • Shell Jobs implemented with any scripting language are supported, provided that the script interpreter, e.g. Python, Ruby etc., is available from on the machine on which the job is executed on.
    • API Jobs are supported for a number of scripting languages to which the JobScheduler Job API is exposed:
      • JavaScript (Nashorn)
      • JScript (Microsoft)
      • Perl (PerlScript)
      • PowerShell (Microsoft)
      • VBScript (Microsoft)
    • JavaScript is available from the Java Virtual Machine (JVM) implementation provided by the Nashorn JavaScript Engine.
    • The scripting languages JScript and VBScript scripting languages are only supported by the Windows JVM on 32bit systems only, see as described in VBScript Jobs. The ScriptControl:VBScript job type introduced with JobScheduler release 1.10 acts as a replacement for VBScript jobs.
  • JS7 includes support for the following scripting languages:
    • The option to run of running Shell Jobs for any scripting language for which the interpreter is available remains unchanged.
    • For Java and JavaScript Jobs the JS7 - Job API is offered.
    • JS7 - JavaScript Jobs are supported for scripting languages available with the GraalVM such asbased on use of the Oracle® GraalVM with JS7 Agents starting from Release 2.5.4 and Release 2.6.1.
    • At the same time JS7 API Java/JavaScript Jobs do not provide the same capabilities as API Jobs for JS1. JS7 API JVM Jobs are limited by design to receive receiving arguments and to return a returning result objects. Actions on inventory objects such as adding orders, stopping jobs etc.  are are available from through the JS7 - REST Web Service API (see below).

Job API migrated to REST Web service

  • The functionality of the JobScheduler Job API has been migrated to the JS7 - REST Web Service API.
  • However, a number of differences have to be considered:
    • The JS1 Job API allows to modify modification of objects of the current job such as the current task or order.
    • The JS7 REST Web Service API allows modification of any objects , but requires to identify objects such as jobsat configuration level that includes deploying modified objects.
    • A number of operations of the JS1 Job API are not available with JS7, for example to modify modifying the job configuration on-the-fly , e.g. by by, for example, adding/dropping a setback configuration. This is not considered good practice as such changes to the configuration are deeply buried in script code and are not evident from configuration.
  • A REST Client has to be used by jobs that access the JS7 REST Web Service API. JS7 ships with a REST Client for Java/JavaScript jobs.

Object Changes

Consider changes described in the JS7 - Terminology article.

Jobs and Job Chains

Job Chains in JS1 correspond to JS7 - Workflows:

  • The JS1 knows Jobs and Job Chains that implement a sequence of jobs.
    • A job is an object that can be modified individually and that which can be re-used with a number of job chains.
    • A job is defined to be operated standalone or as a node in a job chain.
    • Any changes to jobs and job chains are implemented immediately in effect.
  • The JS7 is not limited to a sequence of JS7 - Jobs but introduces a number of JS7 - Workflow Patterns which is why the term workflow replaces job chain.
    • A job is only available within a workflow only. A job is not an independent object and it cannot be re-used.
      • While users might consider this a limitation, in fact it guarantees a self-contained and consistent workflow configuration. Consider the fact that a JS1 job could be modified or removed at run-time leaving the job chain in a critical status. The JobScheduler Master had to check for updated versions of a job XML file and had to apply any modifications in good time. If modifications are not consistent, e.g. if the job parameterization was changed, then this would breaks a current currently running order run as any changes to jobs are applied immediately applied.
      • Modifications of to a JS7 job force a workflow to be deployed once again.
      • The JS7 offers bulk operations for jobs to modify The JS7 - Inventory Bulk Operations for Jobs allow modification of a single job and to apply the application of changes to a number of jobs across workflows.
    • There are no JS1 standalone jobs in JS7, instead such jobs are considered to be workflows with a single job.
      • The JS7 - Job Templates replace the JS1 standalone jobs as they allow to centrally update jobs in a number of workflows.
    • Changes to workflows take effect for newly added JS7 - Orders only. The JS7 Controller keeps track of workflow versions as long as an order is assigned the workflow.
      • At the point of time of submission an order is associated with the current version of a workflow.
      • If the workflow changes later on then this has no impact on submitted orders, but affects will only affect new orders only that are submitted after this change.
      • This feature can be used to target changes of to a workflow to a future date. Existing orders can be cancelled and re-submitted to be assigned the current version of a workflow from the Daily Plan view.

Orders

Orders in JS1 correspond to JS7 - Schedules:

  • JS1
    • The JS1 makes use of uses Orders that include a run-time definition. This means the order recalculates its next start time after execution. At the same time this includes that e.g. means that, for example, suspended orders will not calculate a next start time and therefore will not be executed for future dates as long as they are suspended or blocked , e.g. by by, for example, stopped jobs or job nodes.
    • JS7 cyclic orders are represented as a single order that recalculates its next start time after completion of the last cycle. This includes to use a relative cycle interval that counts from the completion of a previous order run to the start of the next order run.
    • JS1 orders can use Schedules that present run-time definitions shared by a number of orders and jobs
    • JS1 orders carry Parameters. If a parameter is changed then the order has to be changed, i.e. the change is valid for any future executions of an the order starting from the point in time of the change.
  • JS7
    • The JS7 creates individual JS7 - Orders for each execution of a workflow.
      • This allows to modify modification of orders individually from the JS7 - Daily Plan, . - for example to modify the start time and parameters for individual days only.
      • Assume a daily order with a single start time: it will execute each day at the given time independently from of the fact if the order for the previous day has been completed or , is suspended or blocked.
    • A JS7 cyclic order is presented from in the Daily Plan view as a single order. However, however, the cyclic order is submitted to a Controller as an individual order per cycle. This includes means that each execution of an the order cycle is independent from of its predecessor and it includes . It also means that there cannot be a start time relative to completion of the predecessor cycle.

    • JS7 - Schedules correspond to JS1 Orders: a schedule combines a workflow reference, a run-time definition and variables (parameters). From a given schedule the The Daily Plan generates individual orders for each day and start time from a given schedule.

    • The start time times and variables of a JS7 order orders can be modified individually for each order from the Daily Plan view. If the start time or variables from a schedule are changed then this applies to any future orders created with the schedule, it does not apply to existing orders.
      • Note that orders can be submitted days or weeks in advance of their start time by use of using the JS7 - Daily Plan Service.
      • This allows to target changes to a schedule for to be targeted to a specific date. Users can cancel and re-submit existing orders or re-create the Daily Plan if they want changes to a schedule to be in effect starting start from a specific date.

License Model Changes

The JobScheduler and YADE products are available under a dual license model that offers customers the choice between open source licenses and commercial licenses.

  • JS1
    • The license model includes running any number of parallel jobs with Masters and Agents if used with the commercial license.
    • Users of the open source license can run any number of parallel jobs with Masters and are limited to running one task at a time with Agents.
  • JS7
    • JS7 introduces a new license model: for details see the JS7 - License article.
    • Users of the open source license can run any number of parallel jobs with JS7 Agents.
    • The same source code and therefore the same feature functionality is available with both license models. The one exception is the operational feature of clustering JS7 products for high availability which is a commercially available feature for license holders.
  • JS1 + JS7
    • The SOS is committed to open source which is about free software, in the sense of freely available source code.
    • Customers with commercial licenses benefit from additional Support Options and Services.