Page History

Scope

• YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba JCifs, SFTP protocol implemented by JSch library.
• Data providers can make use of individual Java properties, e.g. 
  • for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer
  or
  • for SFTP to specify packet compression to zlib, compression level 9 or
  Cipher
  • specific ciphers such as blowfish-cbc,
  aes192
  • aes128-ctr etc.
• YADE allows to specify Java property System Property files at a global level and individually per file transfer fragment.

  • Jira
    server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-400

  • Java properties are applied as System Properties by YADE.
• YADE allows to specify Java Application Property files as configuration files that are applied individually per file transfer fragment.

  • Jira
    server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key YADE-403

  • Java properties are applied as Application Properties from configuration files. Such properties are specific for a data providerProperties from Java property files are applied as system properties by YADE.

YADE Java System Property

...

Files

YADE Configuration by System Property

...

Files

The System property is a global property Java properties configured using the system_property_files options are global properties available to all protocol fragments and transfer profiles. The SystemPropertyFiles system_property_files option can be assign assigned a semicolon separated list of property files. The A property files file can contain on one or more properties as key value pairs.

Example

Sample System Properties

Following are two examples of such  system property files that contain global Java properties for a data provider. One system property file can also contain properties for multiple data providers. At run-time the YADE will automatically apply applicable properties to the respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider. 

• sftp_system.properties

  Code Block
  language bash title

...

• System Propery File sftp_system.proprerties

  # SFTP properties known_hosts=/home/gollum.sos/.ssh/known_hosts

...

• remote_exec=true

• smb_system.properties

  Code Block
  language bash title System Propery File smb_system.properties
  # SMB properties jcifs.smb.client.useExtendedSecurity=false

Sample YADE Configuration

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties

[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host                                = uk.sos
port                                = 22
user                                = london
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host                                = japan.sos
user                                = tokyo
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec                           = TEST-M-1111.TXT
source_protocol                     = local
source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation                           = copy
file_spec                           = TEST-M-2222.TXT
source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

Explanations

• The YADE settings consist of the following types of sections:
  1. Globals section 
  2. Protocol Fragment sections
  3. Transfer Profile sections
• In the sample yade_settings.ini file, under the globals section the system property files are configure as semicolon separated list of files by the option system_property_files.
• The properties defined in the files sftp_system.properties and  smb_system.properties will be available to all protocol fragments.

• The known_hosts and remote_exec 

Example

Description 

• The YADE profile has three sections 1. globals section 2. Protocol sections and 3. Transfer profile sections.
• The system properties configure under globals section with option system_property_files will be available to all the protocol fragments and transfer profiles.
• In following example the location of hashed known hosts file is configured as name value pair in the file hashed_known_hosts.ini.

• The known_hosts system property will be available and applied to all the fragments and profile using protocol SFTP data provider e,  e.g. to the protocol fragments  fragments protocol_fragment_sftp@sftp-uk.sos and  and protocol_fragment_sftp@sftp-japan.sos.

YADE Java Application Property Files

YADE

...

Confguration by Application Property Files

The protocol fragment specific Java properties can be configured using the configuration_files option at protocol fragment level. The configuration_files option can be assigned a semicolon separated list of property files. A property file can contain one or more properties as key value pairs.

Example

Sample Application Properties for Data Provider

• Code Block
  language bash title SFTP packet compression configuration sftp_zlip_compression.properties
  # ssh packet compression compression.s2c=zlib@openssh.com,zlib,none compression.c2s=zlib@openssh.com,zlib,none compression_level=9

• Code Block
  language bash title SFTP Cipher configuration sftp_cipher_aes128-ctr.properties
  # ssh transfer cipher cipher=aes256-ctr

Sample YADE Configuration

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/hashed_known_hosts.ini
sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties
 
[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host                                = uk.sos
port                                = 22
user                                = london
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa
configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp_zlib_compression.properties
 
[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host                                = japan.sos
user                                = tokyo
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa
configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp_cipher_aes128-ctr.properties

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec                           = TEST-M-1111.TXT

source_protocol                     = local
source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel

target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /home/london/data/to_london/
 
[CopySFTPJapanSOSCopySFTPJapanSOS2Local]
operation                           = copy
file_spec                           = TEST-M-2222.TXT

source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/

target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

YADE System Property Configuration

Explanations

• The global system property file will be applicable as explained in the previous chapter on YADE Java System Property Files. 
• In the YADE Settings sample two configuration files are created, both of them include configuration properties for the SFTP data provider;
  1. sftp_zlip_compression.properties :  effects packet compression during transfer by SFTP data provider.
  2. sftp_cipher_aes128-ctr.properties : effects Cipher used by SFTP data provider.

• The sftp_zlip_compression.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using the configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos, e.g. CopyLocal2SFTPUKSOS.

• Similarly the sftp_cipher_aes128-ctr.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using the configuration_files option, thus the cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos, e.g. CopySFTPJapanSOS2Local.

• At run-time the YADE will automatically apply applicable properties to respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

References

Change Management References