Introduction

JS7 is a rewrite from scratch of the JobScheduler components available with branch 1.x (JS1). This includes that existing *.xml files for scheduling objects have to be migrated.

For general information see JS7 - Migration of JobScheduler 1.x Job Configuration.
For capabilities of the Converter see JS7 - Migration of JobScheduler 1.x - Converter Capability Reference.

Converter

With Release 2.4.0 a Converter becomes available to migrate JS1 *.xml files of scheduling objects to JS7:

JOC-1318 - Getting issue details... STATUS

The Converter reads existing JS1 *.xml files and creates .json files that are added to a .tar.gz or .zip archive for import into JS7, see JS7 - Inventory Export and Import.

Prerequisites

The Converter can be used for the following platforms.

The Converter Java classes can be used with Java JRE or JDK 1.8, 11, 17.
The Converter Start Script is available for
- Linux, MacOS® and AIX® using bash, dash, ksh and zsh shells.
- Windows 7, 8, 10, 11, Windows Server

Download

Find the Converter for download from JS7 - Download.

After extraction of the .tar.gz archive for Unix or .zip archive for Windows find the following files:

js7_convert_js1.sh | .cmd (Converter Start Script)
js7_convert_js1.config (Converter Configuration File)
lib (Directory for Java Classes)

Usage

Invoking the Converter Start Script without arguments displays the usage clause:

Converter Start Script: js7_converter.sh | .cmd

Usage: js7_convert_js1.sh | .cmd [Options]

  Options:
    --input-dir=<location of input directory>                       | required argument
    --output-dir=<location of output directory>                     | default: ./output
    --report-dir=<location of report directory>                     | default: ./report
    --archive=<location of resulting .zip archive for JS7 import>   | default: ./js7_converted_js1.tar.gz | .zip
    --config=<location of config file>                              | default: ./js7_convert_js1.config
    --help                                                          | displays usage

Explanation:

Options
- --input-dir
  - Specifies the input directory of JS1 *.xml files for scheduling objects. Such files typically are located in the SCHEDULER_DATA/config/live folder. If you do not want to run the converter directly on the machine in which the *.xml files are located then you can copy the live folder or any sub-folder to a Unix or Windows machine on which to run the converter. The operating system of JobScheduler Master using the *.xml files is independent from the operating system of the converter.
  - The Converter recursively traverses the input directory.
- --output-dir
  - Specifies the output directory to which converted .*json files are written. For a number of *.xml files there is a corresponding *.json file.
  - For each sub-directory of the input directory the corresponding sub-directory is created in the output directory.
  - On start-up the Converter removes existing files and sub-directories from the output directory.
- --report-dir
  - The Converter will report to the specified directory by use of a number of .csv files the extent to which the conversion is successful:
    - parser_summary.csv: reports the number of objects found in the input directory and sub-directories.
    - parser_analyzer.csv: includes debugging information.
    - converter_summary.csv: reports the number converted objects per object type such as workflows and scheduled.
    - converter_warnings.csv: includes warnings about XML elements that could not be perfectly converted.
- --archive
  - Specifies the path to an archive file with the .tar.gz or .zip extension that will be created by the Converter. Depending on the file extension the related archive format will be used on any OS platform.
  - The archive includes the *.json files of the output directory and can be used for later import into JS7, see JS7 - Inventory Export and Import.
  - An existing archive file will be overwritten.
- --config
  - Specifies the location of the Converter Configuration File.. This file is required for the Converter to run. It includes a number of settings that can be adjusted.

Examples

Unix

Example for running the Converter Start Script for Unix with minimum Arguments

./js7_convert_js1.sh \
    --input-dir=/var/sos-berlin.com/jobscheduler/config/live

# reads *.xml files from the Master's "live" folder and creates the corresponding output directory hierarchy in the "output" sub-directory of the working directory;
# reports are written to the "reports" sub-directory of the working directory, the resulting "js7_converted.tar.gz" archive file is written to the working directory.

Example for running the Converter Start Script for Unix with more Arguments

./js7_convert_js1.sh \
    --input-dir=/var/sos-berlin.com/jobscheduler/config/live \
    --output-dir=/tmp/output
    --report-dir=/tmp/report
    --archive=/tmp/js7-import.zip

# reads *.xml files directly from the Master's "live" sub-directory and creates the corresponding directory hierarchy in "/tmp/output";
# report files are written to "/tmp/report", the resulting archive file is written to "/tmp/js7-import.zip".

Windows

Example for running the Converter Start Script for Windows with minimum Arguments

js7_convert_js1.cmd --input-dir=C:\ProgramData\sos-berlin.com\jobscheduler\config\live

@rem reads *.xml files from the Master's "live" folder and creates the corresponding directory hierarchy in the "output" sub-directory of the working directory;
@rem reports are written to the "reports" sub-directory of the working directory, the resulting "js7_converted.zip" archive file is written to the working directory.

Example for running the Converter Start Script for Windows with more Arguments

js7_convert_js1.cmd ^
    --input-dir=C:\ProgramData\sos-berlin.com\jobscheduler\config\live ^
    --output-dir=C:\tmp\output ^
    --report-dir=C:\tmp\report ^
    --archive=C:\tmp\js7-import.zip

@rem reads *.xml files directly from the Master's "live" sub-directory and creates the corresponding directory hierarchy in "C:\tmp\output";
@rem report files are written to "C:\tmp\report", the resulting archive file is written to "C:\tmp\js7-import.zip".

Configuration

The Converter is configured from a properties file that can be specified with the --config option when invoking the Converter. By default he file js7_convert_js1.config is assumed that ships with the Converter.

Users should adjust the configuration to their needs, for example, to map Agents and Calendars.

Download: js7_convert_js1.config

Default Configuration from js7_convert_js1.config file

# Generate
;generateConfig.workflows                        = false
;generateConfig.agents                           = false
;generateConfig.jobTemplates                     = false
;generateConfig.schedules                        = false
;generateConfig.calendars                        = false
;generateConfig.locks                            = false
;generateConfig.cyclicOrders                     = true

# JobScheduler 1.x
;js1.jobStream.generateJobFileIfNotExists        = true

# Parser
parserConfig.excludedDirectoryNames             = .sos-templates;.svn;.configuration
parserConfig.excludedDirectoryPaths             = sos/

# Workflow
;workflowConfig.defaultTimeZone                  = Etc/UTC

# Workflow Job
;jobConfig.jitl.logLevel                         = INFO
# default: #!/bin/bash
;jobConfig.shell.unix.defaultShebang             = #!/bin/sh
# default: @@findstr/v "^@@f.*&" "%~f0"|pwsh.exe -&goto:eof
;jobConfig.shell.windows.powershellShebang       = @@findstr/v "^@@f.*&" "%~f0"|powershell.exe -&goto:eof
;jobConfig.forcedGraceTimeout                    = 15
;jobConfig.forcedParallelism                     = 1
;jobConfig.forcedFailOnErrWritten                = false
jobConfig.forcedV1Compatible                    = true

# Agent
;agentConfig.forcedControllerId                  = js7  
;agentConfig.defaultControllerId                 = js7
;agentConfig.forcedAgent                         = {"platform":"WINDOWS","standaloneAgent":{"agentName":"primaryAgent","url":"http://localhost:4445"}} 
;agentConfig.forcedAgent                         = examples/agent_standalone.json  
;agentConfig.defaultAgent                        = examples/cluster_agent.json
;agentConfig.mappings                            = js1_agent1=examples/agent_standalone.json; \
                                                   js1_agent2=examples/cluster_agent.json
                                                   
# Mock
;mockConfig.jitl.mockLevel                       = INFO
# INFO      Log arguments and always end successfully
# ERROR     Log arguments and fail if required parameters are missing
;mockConfig.shell.unixScript                     = $HOME/MockScript.sh
;mockConfig.shell.windowsScript                  = %UserProfile%\MockScript.cmd

# Schedule
;scheduleConfig.forcedWorkingDayCalendarName        = AnyDays
;scheduleConfig.forcedNonWorkingDayCalendarName     = AnyDays
;scheduleConfig.defaultWorkingDayCalendarName       = AnyDays
;scheduleConfig.defaultNonWorkingDayCalendarName    = AnyDays   
;scheduleConfig.defaultTimeZone                  = Etc/UTC
scheduleConfig.planOrders                       = true
scheduleConfig.submitOrders                     = true

# Calendar
;calendarConfig.forcedFolder                    = Calendars

Settings

Section: Generate

Setting	Default	Explanation
`generateConfig.workflows`	`true`	Specifies that workflows should be converted.
`generateConfig.agents`	`true`	Specifies that Agent configurations should be created.
`generateConfig.jobTemplates`	`true`	Specifies that jobs used in multiple job chains will be converted to the JS7 - Job Template.
`generateConfig.schedules`	`true`	Specifies that schedules should be converted.
`generateConfig.calendars`	`true`	Specifies that calendars should be converted.
`generateConfig.locks`	`true`	Specifies that Resource Locks should be converted.
`generateConfig.cyclicOrders`	`false`	Specifies that cyclic start times of JS1 orders and schedules will be converted to the JS7 - Cycle Instruction. Consider chapter Cyclic Workflows vs. Cyclic Orders.

Section: Parser

Setting Default Explanation

parserConfig.excludedDirectoryNames

The JS1 live folder includes a number of hidden sub-directories that should not be converted. Such directory names typically start with a dot as in .sos-templates, .configuration.

If more than one directory is specified then they have to be separated with a semicolon ";".

parserConfig.excludedDirectoryPaths

Specifies a number of paths in the JS1 live folder that should not be considered for conversion. Typically the live/sos folder holds housekeeping jobs that are not required for JS7.

If excluded paths are specified then this includes not to consider any sub-directories recursively. It is not required to specify individual sub-directories with the parserConfig.excludedDirectoryNames when using excluded paths.

Section: Workflow

Setting	Default	Explanation
`workflowConfig.defaultTimeZone`	`Etc/UTC`	Specifies the default time zone that is added to converted workflows. This time zone for example is applied to JS7 - Admission Times for Jobs. See List of tz database time zones
`jobConfig.jitl.logLevel`		Specifies the value of the `log_level` job argument of JS7 - Job Templates, see JS7 - JITL Common Variables. Allowed values include: `INFO`, `DEBUG`, `TRACE`. If no log level is specified then JVM Jobs behave as if no `log_level` job argument is used.
`jobConfig.forcedGraceTimeout`	`15`	Specifies the grace timeout after which a running job is terminated if the cancel/force operation is used, see JS7 - Job Instruction.
`jobConfig.forcedParallelism`	`1`	Specifies the number of parallel tasks that a job can be running for, see JS7 - Job Instruction.
`jobConfig.forcedFailOnErrWritten`	`false`	Specifies the handling of job output to the stderr channel. If set to `true` then output to stderr is considered an error, otherwise such output is logged and no error is raised, , see JS7 - Job Instruction.
`jobConfig.forcedV1Compatible`	`false`	JS7 workflows can be operated in a JS1 compatibility mode. If set to `true` then for order variables and workflow variables in shell jobs respective environment variables are created that are prefixed with `SCHEDULER_PARAM_` followed by the uppercase variable name. for job arguments of shell jobs accordingly environment variables are created. Consider that environment variables are not added by the Converter but are created on-the-fly by the Controller and Agents.

Section: Agent

Settings Default Explanation

agentConfig.forcedControllerId Forces use of the given ControllerId for all generated agents.

agentConfig.defaultControllerId This setting specifies the ControllerId for generated agents if the Master ID in JS1 (process_class.spooler_id) is empty.

agentConfig.forcedAgent

Forces use of the given Agent for all converted jobs.

Agent definition:

Expects a JSON value.
The JSON can be provided as:
- JSON text
  or
- JSON include file (relative to the properties file)

JSON elements:

Name	Default	Explanation
`platform`	`UNIX`	Forces use of the UNIX or WINDOWS platform.
`subagentClusterId`		Forces use the given `subagentClusterId` for converted jobs when an agentCluster is used.
`standaloneAgent`		See agent-schema.json.
`agentCluster`		See clusterAgent-schema.json.

Examples:
- {"platform":"UNIX"}
  - Forces use of the UNIX platform for all converted jobs.
- {"standaloneAgent":{"url":"http://localhost:4445"}}
  - Forces use of the standalone Agents in all converted jobs with the given url.
- {"platform":"UNIX", "standaloneAgent":{"agentName":"forced_agent", "url":"http://localhost:4445"}}
  - Forces use of a standalone Agent forced_agent in all converted jobs with the given url.

agentConfig.defaultAgent

Name	URL
`default_agent`	`http://localhost:4445`

Without an Agent being specified the JS1 executes jobs with the Master. In JS7 an Agent has to be used. This setting specifies an Agent that is used for jobs that are executed with a Master in JS1.

The syntax for assignments: see agentConfig.forcedAgent Agent definition.

agentConfig.mappings

By default the Converter handles the mapping of Agent Names found in JS1 process classes (<process_class>) like this:

Agents are determined by the unique URL assigned the Agent in existing process classes.
The name of a process class is mapped
- to a Standalone Agent if it includes a single Agent
- to a Cluster Agent if it includes more than one Agent

The syntax for mapping includes:

<js1-agent-name>=<Agent definition>

<js1-agent-name> is the name of the JS1 Agent.
<Agent definition> see agentConfig.forcedAgent Agent definition..

Example:

js1_agent1={"platform":"UNIX", "standaloneAgent":{"agentName":"primaryAgent", "url":"http://unix:4445"}};js1_agent2={"platform":"WINDOWS", "standaloneAgent":{"agentName":"secondaryAgent", "url":"http://win:4445"}}

The process class js1_agent1 is mapped to the JS7 Agent with the name primaryAgent that is operated for Unix.
The process class js1_agent2 is mapped to the JS7 Agent with the name secondaryAgent that is operated for Windows.

or

js1_agent1=examples/agent_standalone.json; js1_agent2=examples/cluster_agent.json

The process class js1_agent1 is mapped to the JS7 Agent defined in the examples/agent_standalone.json file.
The process class js1_agent2 is mapped to the JS7 Agent defined in the examples/cluster_agent.json file.

Any number of mappings are specified separated by a semicolon. This setting can be split across a number of lines by use the "\" line continuation character at the end of each line that should be continued.

Section: Mock

Setting Default Explanation

mockConfig.jitl.mockLevel

Mock levels are used for dry runs of the conversion for JS7 - Job Templates (JITL JVM Jobs). If a mock level is specified then JVM Jobs will not be executed but will log and will optionally check arguments only.

The following values can be specified for this setting:

INFO: arguments are logged, execution of JVM Jobs is considered successful.
ERROR: arguments are logged, required arguments of JVM Jobs are checked and errors are raised in case of missing arguments.

Without this setting being specified the JITL JVM Jobs will be executed from the converted jobs.

mockConfig.shell.unixScript

Mock scripts are used for dry runs of the conversion. The job scripts of converted Shell Jobs are replaced by the call to a mock script for Unix platforms. Consider that mock scripts have to be provided by the user and have to be available for Agents at run-time.

This setting specifies the path to a mock script for Unix, for example, $HOME/MockScript.sh

A mock script for Unix can look like this:

Example for Mock Script on Unix

#!/bin/sh
echo "------------------------------------------------------------"
echo "Start Unix Mock Script: $0"
echo "------------------------------------------------------------"
echo "Arguments: $*"
echo "------------------------------------------------------------"

If no mock script is specified then the original JS1 job script is used for the converted JS7 job.

mockConfig.shell.windowsScript

Mock scripts are used for dry runs of the conversion. The job scripts of converted Shell Jobs are replaced by the call to a mock script for Windows platforms. Consider that mock scripts have to be provided by the user and have to be available for Agents at run-time.

Specifies the path to a mock script for Windows, for example, %UserProfile%\MockScript.cmd

A mock script for Windows can look like this:

Example for Mock Script on Windows

@echo off
@echo ------------------------------------------------------------
@echo Start Windows Mock Script: %0
@echo ------------------------------------------------------------
@echo Arguments: %*
@echo ------------------------------------------------------------

If no mock script is specified then the original JS1 job script is used for the converted JS7 job.

Section: Schedule

Setting	Default	Explanation
`scheduleConfig.forcedWorkingDayCalendarName`	`AnyDays`	With the JS1 use of calendars is optional, with the JS7 use of JS7 - Calendars is required. This setting forces use of the same calendar for working days with any schedules that are created. By default a calendar with the name `AnyDays` is assumed. Users have to create a calendar with this name prior to running the Converter.
`scheduleConfig.forcedNonWorkingDayCalendarName`		This setting forces use of the same calendar for non-working days with any schedules that are created. Use of a calendar for non-working days is optional. Users have to create a calendar with this name prior to running the Converter.
`scheduleConfig.defaultWorkingDayCalendarName`	`AnyDays`	This setting specifies a default calendar for working days for schedules that do not specify a calendar on their own. By default a calendar with the name `AnyDays` is assumed. Users have to create a calendar with this name prior to running the Converter.
`scheduleConfig.defaultNonWorkingDayCalendarName`	`AnyDays`	This setting specifies use of a default calendar for non-working days with schedules that do not specify a calendar on their own. Use of a calendar for non-working days is optional. Users have to create a calendar with this name prior to running the Converter.
`scheduleConfig.defaultTimeZone`	`Etc/UTC`	With the JS1 a schedule optionally specifies a time zone. With the JS7 a time zone is required for scheduled. This setting specifies the default time zone that is applied if a JS1 schedule does not specify a time zone. See List of tz database time zones
`scheduleConfig.planOrders`	`false`	Orders are automatically planned for the JS7 - Daily Plan.
`scheduleConfig.submitOrders`	`false`	Orders are automatically submitted to Controllers for the JS7 - Daily Plan.

Section: Calendar

Setting Default Explanation

calendarConfig.forcedFolder

Root folder

Specifies the folder where the calendars are generated.

Example:

calendarConfig.forcedFolder = Calendars

Section: JS1

This section includes settings specified for JobScheduler 1.x.

Section: JobStream

Setting Default Explanation

js1.jobStream.generateJobFileIfNotExists

false

This refers to a situation where there is only a JSON definition available for a job stream.

Note: the converter should be restarted if the JS1 <job name>.job.xml files have been created.

Recommendations

It is recommended to create the following objects in the JS7 inventory prior to conversion:

Agents
- At least one Agent has to be available. In the JS1 jobs can be executed with a Master, in JS7 an Agent is required.
- The Converter offers to map Agent Names from JS1 process classes to JS7 Agents. Such Agents have to be available in the JS7 prior to conversion. Users can register Agents as Standalone Agents and as Cluster Agents, see JS7 - Agent Management.
Calendars
- At lease one working day calendar has to be available. The calendar can be stored in an arbitrary folder. The name of the calendar can be specified with the scheduleConfig.forcedWorkingDayCalendarName and scheduleConfig.defaultWorkingDayCalendarName settings or it can default to AnyDays.
- If the calendar specified by converted schedules is not available in the JS7 then imported schedules become invalid. They can be individually assigned a calendar in the JS7 inventory.

Delimitations

At the time of writing the following configuration elements are not supported:

generateConfig.cyclicOrders

Support for the indicated configuration elements will be added before August 2022.

Further Information

JS7 - Migration of JobScheduler 1.x - Converter Capability Reference

Space shortcuts

Page tree

Introduction

Converter

Prerequisites

Download

Usage

Examples

Unix

Windows

Configuration

Settings

Section: Generate

Section: Parser

Section: Workflow

Section: Agent

Section: Mock

Section: Schedule

Section: Calendar

Section: JS1

Section: JobStream

Recommendations

Delimitations

Further Information

Space shortcuts

Page tree

JS7 - Migration of JobScheduler 1.x - Converter Usage

Introduction

Converter

Prerequisites

Download

Usage

Examples

Unix

Windows

Configuration

Settings

Section: Generate

Section: Parser

Section: Workflow

Section: Agent

Section: Mock

Section: Schedule

Section: Calendar

Section: JS1

Section: JobStream

Recommendations

Delimitations

Further Information