...

• Sizing is an important step when planning the installation and operation of JS7.
• The following guidelines are recommendations (best practices) that have require to be mapped to the requirements of your environment.
• We recommend to decide SOS recommends deciding on the desired target architecture in a first step and then to determine determining the sizing accordingly.

Architecture

• First an architecture decision of all, a decision about the architecture has to be taken:
  • Use of JOC Cockpit and Controller standalone instances
  • Use of JOC Cockpit and Controller high-availability cluster instances
  • For details see see the JS7 - System Architecture article.
• Then the scheduling modes have to mode should be identified:
  • Use of the of JS7 AgentAgents
  • Use of JS7 - Agentless Scheduling

Scalability

The JS7 adjusts to both vertical and horizontal scaling, see the JS7 - Performance article:

• Vertical Scalability
  • There are no built-in limits as to the overall number of jobs and number of parallel job executions per Controller and Agent.
  • A single Controller is tested for some 50 000 parallel tasks, a single Agent is tested for approx. 15 000 parallel tasks.
  • Adding more power to a single machine (CPU, Memory, SSD Disks) will improve performance as the products will make use of all available cores.
• Horizontal Scalability
  • Splitting the job load across a number of Agents will push the number of parallel tasks.

Users have a choice to size for vertical or horizontal scalability.

Sizing

Server Resources

• The following indicators should be considered:
  • The overall number of workflows and jobs that will be available.
  • The number of jobs that will be executed in parallel.
  For general information on resource consumption of JobScheduler Master and Agent see CPU and Memory Usage
• Number of Jobs
  • Small environments are assumed to use < 1 000 jobs, medium sized environments use < 20 000 jobs, larger environments use larger numbers.
• Parallelism of Jobs
  • Small environments run < 100 parallel jobs, medium sized environments run < 1 000 parallel jobs, larger environments run larger numbers of jobs.

Basic Recommendation

The recommendation indicates dedicated resources available to JS7 products. Resource usage by additional software running on a machine come on top.

Overall number of jobs

• Impact
  • JobScheduler JS7 is designed to operate up to 20 1 000 000 overall job objects. For a higher number of jobs or with respect to the sharing of organisational responsibilities it is recommended to However, to share duties users might split the load across multiple JobScheduler JS7 instances.
  • A high number of orders, workflows and jobs affects the responsiveness of the JOC Cockpit GUI that shows the objects for jobs, job chains, orders etc.. Performance of the JOC GUI can be increased by use of the Jetty web server that comes bundled with JobScheduler instead of the built-in web server.in displaying the objects in question.
• Recommendations
  • For small and medium sized environments no specific measures are required.
  • For larger environments a number of measures are applied that are tailored to the user's needs
  Recommendations
  • JobScheduler is designed for re-usability of jobs and job chains. The same jobs and job chains can be started with different parameter sets and can therefore be re-used.
  • Check how you would want to implement or migrate your jobs and job chains. SOS provides Consulting Services to assist you in designing your jobs and job chainsworkflows and jobs for performance.

...

CPU

• Impact
  • Starting from your architecture decision the distribution of jobs across Master and Agent instances should be considered.
    • Jobs that are executed on Agents do not result in direct CPU and memory requirements on the Master.
    • Jobs that are executed agentless by SSH require a process that is executed on the Master and a process on the SSH server for execution of the job script.
  • For each job executed in parallel a JobScheduler instance the following resources are required:
    • A Java Virtual Machine is started that requires at least 32 MB memory. When using API jobs or JITL jobs then this value could be increased to 64 MB depending on the nature of the job.
    • The scripts and programs that you execute will cause individual requirements concerning CPU and memory.
    • When running jobs for database processing then the load will be in the database and not in the JobScheduler instance. CPU and memory requirements are usually stable for such jobs independently from the size or duration of transactions that are performed by such jobs.
• Recommendations
  • Calculate at least 32 MB memory for each API job or JITL job, e.g. 7 GB memory for approx. 200 parallel job executions. Shell jobs wiithout monitor scripts would not exceed about 20 MB memory each.
  • Consider to share the load by use of Agents that are executed on different servers. There is no hard limit concerning the number of Agents that can be operated with a Master, however, we recommend not to exceed 1024 Agent instances with one Master.

CPU

• • The JOC Cockpit will use a small amount of CPU during idle times due to the fact that a number of background processes are running, see the JS7 - Services article. When load is increased by parallel users or by parallel use of the JS7 - REST Web Service API then more threads will be used that map to processes and cores.
  • The Controller and Agent will hardly use any CPU power during idle times. The Controller and Agent handle thread pools in way that balances CPU consumption across all available cores.
  Impact
  • JobScheduler has a low footprint when it comes to CPU usage. Practically no CPU usage applies to idle jobs and monitored directories.
• Recommendations
  • Do not use a single core CPU system. Situations could occur when individual job scripts jobs might misbehave and consume more CPU than expected. A dual multi-core CPU system allows to have means that process resources ready will be available in a situation when a single core CPU is blocked. The more cores that are available the better the performance results will be.

Memory

• Impact
  • The number of parallel orders and jobs increases memory consumption of an active Controller instance and the relevant Agents.
    • The JOC Cockpit can be operated from 512 MB heap size. This value should be increased if more users are working in parallel with JOC Cockpit or if a larger number of orders are available in the JS7 - Daily Plan.
    • The Controller can be operated starting from 256 MB memory. Higher parallelism of orders and jobs will require higher values, e.g. for 100 000 orders approx. 4 GB will be required.
    • The
  • The JobScheduler Master requires about 200 MB memory.
  • The JobScheduler Universal
    • Agent has a footprint of about 100 MB memory. As with the Controller higher parallelism of orders and jobs will push this value. As a rule of thumb 2 GB main memory should be sufficient to execute 5 000 jobs in parallel on an Agent.
  • Refer to the JS7 - FAQ - Which Java Options are recommended article for default values and details how to specify memory consumption.
• Recommendations
  • Calculate at least 32 MB memory for each job, e.g. 7 GB memory for approx. 200 parallel job executions500 MB for JOC Cockpit for 5 to 10 parallel users. Calculate 2 GB each for a Controller and Agent if < 50 000 task executions per day and < 5000 parallel task executions are performed. You can reduce the values for Controller and Agent to 500 MB if < 10 000 task executions per day and < 1000 parallel task executions are performed.

Initial Installation

• File System
  • The size of the installed files should not exceed approx. 200 MB250 MB for each JOC Cockpit, Controller and Agent.
• Database
  • The database initially can be used with as little as 200 MB tablespace.

Ongoing Operation

File System

• Job related Journal files
  • The overall size of job related files is negligible and would hardly exceed 50 MB.
  • Journal files are used by Controller instances and Agent instances to store information about state transitions and log output. Typically such information is immediately forwarded from an Agent to the Controller and from the Controller to JOC Cockpit. However, in case of high load or outage of a product such as a Controller, then the Agent will continue to execute jobs and store such information in its journal. Journal files therefore can grow, technically to the max. size imposed by the OS. Journal files will shrink after JS7 products are recoupled. For most situations with <100 000 job executions per day and an outage period < 24h it is sufficient to assume 5 GB diskspace for journal files. 
• Log files
  • Log files are by default configured for log rotation to limit diskspace consumption and to restrict the number of files in use, see the JS7 - Log Rotation article for more information.
    
    • All log files are controlled by Log4j2 and allow individual configuration.
    • By default diskspace consumption is limited to 5 GB and log files are retained for 30 days. This applies to each JOC Cockpit, Controller and Agent
  Log files
  • The following indicators affect the estimated size for log file storage:
    • Job frequency: for each job execution a log file is created with approx. 1 KB file size. The log file for a task will be overwritten with each execution of the job. Therefore the number of log files for jobs and tasks will be stable in relation to the overall number of jobs.
    • Job log output: The log output of JobScheduler is restricted to a few lines, however, your job scripts (programs, applications) will create individual log output.
    • JobScheduler log files: JobScheduler writes its own log files including a main log for the lifetime of JobScheduler execution and a debug log for analysis purposes.
    • JobScheduler log levels: When debugging jobs then you could increase the log level and cause JobScheduler to create huge logs. 
    • JobScheduler log rotation: Determine the retention period for log files. JobScheduler provides a houskeeping job to rotate and zip log files. A typical policy on log files could e.g. include to have all log files of the previous week available on disk, to have zipped versions of the log files available for the last three months and to move older log files to some cheaper long term storage.
  • Recommendations
    • It is suggested not to start below 2 that not less than 1 GB disk space is assigned for logs. For a mid-sized environment with e.g. 1000 job executions per day you should not start below 10 GB.Use of more than 5 GB disk space should be required only if the retention period for log files is prolonged.
    • When JS7 products cannot write to a log file then they When JobScheduler cannot write to its log file then it will be blocked from further operation. To prevent such a situation add a disk space monitoring task to your system monitor.

...

• Shared/Dedicated Database Server
  • The JS7 does not require a dedicated Database Server. At the same time in a shared scenario users should be aware of the impact of impacts if the Database Server is being subject to high load.
  • If the JS7 faces insufficient database performance then this will slow down the performance of the JOC Cockpit, preferably the JS7 - History might not be able to report job execution results and log output in good time. Practically speaking, if the job execution history is typically is available within 1-2s after completion of a job then in case the event of high load of job executions and bad performance  of the database execution load and poor database performance this can take longer, maybe minutes. However, such the data will not be lost but are will be added with a delay.
•  Memory
  • Some DBMS allow tables to store tables be stored in memory. This can improve the overall performance for frequently used tables. 
  • Recommendation
    • SOS recommends to leave leaving this subject to your DBA. This is about database optimization and will result in a noticeable effect only in high performance environments.
• Tablespace
  • JS7
    • stores the job execution history in the database,
    • stores the task logs in a zipped format in the database,
    • can be configured to automatically purge the database for after configurable object retention periods of objects, see : see the JS7 - Cleanup Service article.
  • Recommendations
    • Calculate the increase in tablespace size based on the overall number of job executions. Assume 2 KB for job history and task log for each job execution. Add a factor for individual log output of scheduled scripts and programs.
    • A practical approach includes to monitor closely monitoring tablespace growth during the first few months of operation and then to adjust adjusting the configuration of the Cleanup Service to purge for shorter or longer retention periods.
    • As a rule of thumb do not start JS7 operation with a tablespace <250 <200 MB.

Further Resources

• JS7 - Performance
• JS7 - FAQ - Which Java Options are recommended
• JS7 - How To - Apply Java Options
• What size will a database used by JobScheduler have?
• CPU and Memory Usage
• Performance