Page History

Status

colour	Yellow
title	In Progress

Table of Contents

Scope

YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba JCifs, SFTP protocol implemented by JSch library.
Data providers can make use of individual Java properties, e.g.
- for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer
or
- for SFTP to specify packet compression to zlib, compression level 9 or
Cipher
- specific ciphers such as blowfish-cbc,
aes192
- aes128-ctr etc.
YADE allows to specify Java property System Property files at a global level and individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-400
- Properties from Java property files properties are applied as system properties System Properties by YADE.
YADE allows to specify Java Application Property files as configuration files that are applied individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-403
- An application configuration file contains settings that are specific to an app/Java properties are applied as Application Properties from configuration files. Such properties are specific for a data provider.

YADE Java System Property

...

Files

YADE Configuration by System Property

...

Files

The Java properties configured using System Property Files the system_property_files options are global properties available to all protocol fragments. The SystemPropertyFiles system_property_files option can be assign assigned a semicolon separated list of property files. The A property files file can contain on one or more properties as key value pairs.

Example

Sample System Properties

Following are two examples of such system property files that contain global Java properties for a data provider. One system property file can also contain Following is the example of such a System Property File which contains global properties for multiple data providers. The At run-time the YADE will apply automatically apply applicable properties to the respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

sftp_system_proprieties.iniproperties
Code Block
language bash
title SystemProperyFile System Propery File sftp_system_proprieties.inproprerties
# SFTP properties known_hosts=/home/gollum.sos/.ssh/known_hosts remote_exec=true
smb_system_proprieties.iniproperties
Code Block
language bash
title SystemProperyFile System Propery File smb_system_proprieties.iniproperties
# SMB properties jcifs.smb.client.useExtendedSecurity=false

...

Sample YADE Configuration

Description

...

Globals section
ProtocolsFragments section
Transfer Profiles section

...

Code Block

language	bash
title	YADE Settings file with system_property_files yade_settings.ini
collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties

[protocol_fragment_sftp@sftp-

...

YADE Settings

Code Block

language	bash
title	YADE Settings system_property_files
collapse	true

[globals]
historyuk.sos]
protocol                            = sftp
host                = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_proprieties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_proprieties.ini

[protocol_fragment_sftp@sftp-uk.sos]
protocoluk.sos
port                                = 22
user            = sftp
host                   = london
ssh_auth_method            = uk.sos
port        = publickey
ssh_auth_file                       = 22
user${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host         = london
ssh_auth_method                       = publickey
ssh_auth_filejapan.sos
user                         = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol      = tokyo
ssh_auth_method                     = sftp
host         publickey
ssh_auth_file                       = japan.sos
user${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec    = tokyo
ssh_auth_method                       = publickey
ssh_auth_fileTEST-M-1111.TXT
source_protocol                     = local
source_dir   = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec     /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = TEST-M-1111.TXT
source_protocolprotocol_fragment_sftp@sftp-uk.sos
target_dir                      = local
source_dir    = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation                           = /mnt/r2d2/nobackup/data/from_galadriel
target_includecopy
file_spec                      = protocol_fragment_sftp@sftp-uk.sos
target_dir    = TEST-M-2222.TXT
source_protocol                     = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation     sftp
source_include                      = copy
file_spec protocol_fragment_sftp@sftp-japan.sos
source_dir                          = TEST-M-2222.TXT
source/home/tokyo/data/from_tokyo/
target_protocol                     = sftplocal
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_target_dir                          = /homemnt/r2d2/tokyonobackup/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

YADE Configuration Files Configuration

The ConfigurationFiles are properties available only to the protocol fragments they are assigned. The ConfigurationFile option can be assign a semicolon separated list of property files. The property files can contain on or more properties as key value pairs.

Example

...

language	bash
title	SFTP packet compression configuration sftp-zlip-compression.ini

...

to_galadriel

Explanations

The YADE settings consist of the following types of sections:
1. Globals section
2. Protocol Fragment sections
3. Transfer Profile sections
In the sample yade_settings.ini file, under the globals section the system property files are configure as semicolon separated list of files by the option system_property_files.
The properties defined in the files sftp_system.properties and smb_system.properties will be available to all protocol fragments.
The known_hosts and remote_exec system property will be available and applied to all the fragments using protocol SFTP, e.g. to the protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Java Application Property Files

YADE Confguration by Application Property Files

The protocol fragment specific Java properties can be configured using the configuration_files option at protocol fragment level. The configuration_files option can be assigned a semicolon separated list of property files. A property file can contain one or more properties as key value pairs.

Example

Sample Application Properties for Data Provider

Code Block
language bash
title SFTP Cipher packet compression configuration sftp-cipher-aes128-ctr.ini_zlip_compression.properties
# ssh transferpacket ciphercompression cipher=aes128-ctr

Description

The global SystemProperty file will be applicable as described in the previous section YADE System Property Configuration.
In the following example two configuration files are created, both of them has configuration properties for the SFTP data provider;
1. sftp-zlip-compression.ini : effects packet compression during transfer over SFTP.
2. sftp-cipher-aes128-ctr.ini : effects Cipher used by SFTP data provider.
The sftp-zlip-compression.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos e.g. CopyLocal2SFTPUKSOS
Similarly the sftp-cipher-aes128-ctr.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using configuration_files option, thus the Cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos e.g. CopySFTPJapanSOS2Local.

YADE Settings

...

language	bash
title	YADE Settings system_property_files
collapse	true

...

compression.s2c=zlib@openssh.com,zlib,none
compression.c2s=zlib@openssh.com,zlib,none
compression_level=9

Code Block
language bash
title SFTP Cipher configuration sftp_cipher_aes128-ctr.properties
# ssh transfer cipher cipher=aes256-ctr

Sample YADE Configuration

Code Block

language	bash
title	YADE Settings with system_property_files and configuration_files yade_settings.ini
collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties
 
[protocol_fragment_sftp@sftp-uk.sos]
protocol                            =

...

sftp
host                                =

...

uk.sos
port                                = 22
user

...

= london
ssh_auth_method

...

        = publickey
ssh_auth_file                       =

...

${HOME}/.ssh/id_rsa
configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp_zlib_compression.properties
 
[protocol_fragment_sftp@sftp-japan.sos]
protocol

...

             = sftp
host

...

japan.sos
user                                =

...

tokyo
ssh_auth_method                     =

...

publickey
ssh_auth_file

...

= ${HOME}/.ssh/id_rsa
configuration_files

...

          = /home/test/sos-berlin.com/jade_client/configurations/sftp_cipher_aes128-ctr.properties

[CopyLocal2SFTPUKSOS]
operation

...

      = copy
file_spec                           =

...

TEST-M-1111.TXT
source_protocol

...

local
source_dir

...

      = /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir

...

                     = /home/london/data/to_london/
 
[CopySFTPJapanSOS2Local]
operation

...

copy

...

file_

...

spec                           =

...

TEST-M-2222.TXT
source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-

...

japan.sos

...

source_dir                          = /home/

...

tokyo/data/

...

from_

...

tokyo/

...

target_protocol

...

= local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

Explanations

The global system property file will be applicable as explained in the previous chapter on YADE Java System Property Files.
In the YADE Settings sample two configuration files are created, both of them include configuration properties for the SFTP data provider;
1. sftp_zlip_compression.properties : effects packet compression during transfer by SFTP data provider.
2. sftp_cipher_aes128-ctr.properties : effects Cipher used by SFTP data provider.
The sftp_zlip_compression.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using the configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos, e.g. CopyLocal2SFTPUKSOS.
Similarly the sftp_cipher_aes128-ctr.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using the configuration_files option, thus the cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos, e.g. CopySFTPJapanSOS2Local.
At run-time the YADE will automatically apply applicable properties to respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.
= TEST-M-2222.TXT source_protocol = sftp source_include = protocol_fragment_sftp@sftp-japan.sos source_dir = /home/tokyo/data/from_tokyo/ target_protocol = local target_dir = /mnt/r2d2/nobackup/data/to_galadriel

References

Change Management References

...

Space shortcuts

Page tree

Versions Compared

Old Version 9

New Version Current

Key

Scope

YADE Java System Property

Files

YADE Configuration by System Property

Files

Example

Sample System Properties

Sample YADE Configuration

Description

YADE Settings

YADE Configuration Files Configuration

Example

Explanations

YADE Java Application Property Files

YADE Confguration by Application Property Files

Example

Sample Application Properties for Data Provider

Description

YADE Settings

Sample YADE Configuration

Explanations

References

Change Management References