...
The SSH Session Management adds the possibility to end orphaned remote processes started by SSH Jobs or jobs or orphaned JobScheduler tasks from SSH Jobsjobs.
Use Case
What happens if the connection to your remote host breaks while a script is still running? How can the JobScheduler job which started the remote script know about that?
...
You can configure your existing SSH job chain to start the monitoring job chain, which then will monitor the task of your original job chain as well as the processes started on the remote host via SSH.
Anchor | ||||
---|---|---|---|---|
|
...
of Operation
To configure your SSH Job job to use the SSH Session Management you have to configure your SSH Job job and define a cleanup job chain for the cleanup work.
The feature requires the use of the JSch implementation by JCraft. See How To - Usage of the SSH Job (JobSchedulerSSHJob) with JCraft's JSch for more information about configuring your SSH Job job to use the JSch implementation,
...
- if a remote process is running and the JobScheduler Task task is still alive:
- do nothing, the cleanup job chain goes to a setback condition and waits for another start
- if a remote process is running but the JobScheduler Task task is no longer available:
- the cleanup job tries to end the process on the remote host
- if a remote process is no longer available but the JobScheduler Task task is still running:
- the cleanup job ends the Task task immediately
- if a remote process and the JobScheduler Task task are not available anymore
- do nothing, the cleanup job ends
...
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
Configuration of the SSH
...
job to be monitored
A number of additional parameters have to be added to the job configuration before the SSH Job job can be monitored:
runWithWatchdog
...
- format:
<command> \${pid}
- default value:
kill -15 \${pid}
- description:
- The command for terminating a process on the remote machine. This command depends on the OS of the remote host. If the command is not set, the cleanup Job job checks whether the remote host is running on a Linux or on a Windows system and uses the appropriate default commands.
The${pid}
placeholder will be substituted by the cleanup job. Note that the leading $ character has to be escaped with "\".
- The command for terminating a process on the remote machine. This command depends on the OS of the remote host. If the command is not set, the cleanup Job job checks whether the remote host is running on a Linux or on a Windows system and uses the appropriate default commands.
...
- format:
<command> \${pid} \${user}
- default value:
/bin/ps -ef | grep \${pid} | grep \${user} | grep -v grep
- description:
- The command or script checks if the process on the remote host is still running. The cleanup job expects an exitcode = 0 if the process is still running or other than 0 if the process is not available anymore. The command or script depends on the OS of the remote host. If the command is not set, the default command for Linux is used.
The${pid}
and${user}
placeholders will be substituted by the cleanup job. Note that the leading $ character has to be escaped with "\". - Example commands:
/bin/ps -ef
- writes all running processes to stdout on the remote host
| grep \${pid}
- filters the result to show only results containing the given ${pid}
- filters the result to show only results containing the given ${pid}
| grep \${user}
- filters the result to show only results containing the given ${user}
| grep -v grep
- filters the grep command itself
- filters the grep command itself
- The command or script checks if the process on the remote host is still running. The cleanup job expects an exitcode = 0 if the process is still running or other than 0 if the process is not available anymore. The command or script depends on the OS of the remote host. If the command is not set, the default command for Linux is used.
Example: Configuration
...
with JOE
Job
Parameters
Example: XML Configuration
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<job order="yes" stop_on_error="false" title="Launch commands or executable files by SSH"> <description> <include file="jobs/JobSchedulerSSHJob.xml"/> </description> <params> <param name="host" value="[HOST]"/> <param name="port" value="[SSHPORT]"/> <param name="user" value="[USERNAME]"/> <param name="password" value="[PASSWORD]"/> <param name="auth_method" value="password"/> <param name="command_script_file" value="[PATH_TO_SCRIPTFILE]/\test_sleep_90s.sh"/> <param name="runWithWatchdog" value="true"/> <param name="cleanupJobchain" value="kill_jobs/remote_cleanup_test"/> <param name="ssh_job_kill_pid_command" value="kill -9 \${pid}"/> <param name="ssh_job_terminate_pid_command" value="kill -15 \${pid}"/> <param name="ssh_job_get_pid_command" value="echo $$"/> <param name="ssh_job_get_active_processes_command" value="/bin/ps -ef | grep \${pid} | grep \${user} | grep -v grep"/> </params> <script java_class="sos.scheduler.job.SOSSSHJob2JSAdapter" language="java"/> <run_time/> </job> |
Anchor | ||||
---|---|---|---|---|
|
...
of the Cleanup job chain
A cleanup job chain with two jobs has to be configured to process the cleanup of the remote processes or the JobScheduler task.
The cleanup job chain consists of two jobs, one to read the pid of the connected shell from a temporary file on the remote host and one to check if the process or the JobScheduler Task task is still running. The temporary file will be generated automatically and deleted after processing. The second job also ends either the remote process or the JobScheduler task as appropriate.
Job 1: The read-pid-from-temporary-file
...
job
The first job reads the pid from the temporary file on the remote host. If your SSH Job job is configured as described above, the temporary file will have been created automatically on the remote host.
...
After choosing the relevant class name from the list, configure a setback for the job. The setback will be used to restart the job according to the conditions found by the job (described above).
Job 2: The check-and-kill
...
job
The second job checks:
- if the pid determined by the first job is still active
- if the JobScheduler task is still active
- kills the remote process or the JobScheduler task
...
Configure the class of the job in JOE as follows:
Example
...
: XML Configuration
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<job order="yes" stop_on_error="no" title="Launch read pid file command by SSH"> <description> <include file="jobs/SOSSSHReadPidFileJob.xml"/> </description> <script java_class="sos.scheduler.job.SOSSSHReadPidFileJobJSAdapter" language="java"/> <delay_order_after_setback delay="30" is_maximum="no" setback_count="1"/> <delay_order_after_setback delay="0" is_maximum="yes" setback_count="3"/> <run_time/> </job> |
...