How to configure Java property files for data provider

IN PROGRESS

Scope

• YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba, SFTP protocol implemented by JSch library.
• Data providers can make use of individual Java properties, e.g. for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer or for SFTP to specify packet compression to zlib, compression level 9 or Cipher blowfish-cbc,aes192-ctr etc.
• YADE allows to specify Java property files at a global level and individually per transfer fragment.
  • YADE-400 - Getting issue details... STATUS
  • Properties from Java property files are applied as system properties by YADE.
• YADE allows to specify configuration files individually per transfer fragment.
  • YADE-403 - Getting issue details... STATUS
  • An application configuration file contains settings that are specific to an app/data provider.
    
    

YADE Java Property File Configuration

YADE System Property Configuration

The Java properties configured using system_property_files options are global properties available to all protocol fragments. The system_property_files option can be assign a semicolon separated list of property files. A property file can contain on or more properties as key value pairs.

Following are two examples of such  system property files, which contains global Java properties for a data providers. One system property file can also contain properties for multiple data provider. At the run time the YADE will automatically apply applicable properties to respective data provider e.g. SFTP properties to JSch data provider and SMB properties to JCifs data provider. 

• sftp_system_properties.ini

  SystemProperyFile sftp_system_proprerties.in

  # SFTP properties
  known_hosts=/home/gollum.sos/.ssh/known_hosts
  remote_exec=true

• smb_system_properties.ini

  SystemProperyFile smb_system_properties.ini

  # SMB properties
  jcifs.smb.client.useExtendedSecurity=false

Example

Description 

• The YADE profile has three sections
  1. Globals section 
  2. Protocol Fragments section
  3. Transfer Profiles section
• In the following example yade_settings.ini file, under the globals section the system property files are configure as semicolon separated list of files by the option system_property_files.
• The properties defined in the files sftp_system_properties.ini and  smb_system_properties.ini will be available to all the protocol fragments. At the run time the YADE will automatically apply applicable properties to respective data provider e.g. SFTP properties to JSch data provider and SMB properties to JCifs data provider.

• The known_hosts and remote_exec system property will be available and applied to all the fragments using protocol SFTP  e.g. to protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Settings

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_properties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_properties.ini

[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host                                = uk.sos
port                                = 22
user                                = london
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host                                = japan.sos
user                                = tokyo
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec                           = TEST-M-1111.TXT
source_protocol                     = local
source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation                           = copy
file_spec                           = TEST-M-2222.TXT
source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

YADE Configuration Files Configuration

The protocol fragment specific Java properties can be configured using configuration_files option on protocol fragment level . The configuration_files option can be assign a semicolon separated list of property files. A property file can contain on or more properties as key value pairs.

Example

• SFTP packet compression configuration sftp-zlip-compression.ini

  # ssh packet compression
  compression.s2c=zlib@openssh.com,zlib,none
  compression.c2s=zlib@openssh.com,zlib,none
  compression_level=9
  

• SFTP Cipher configuration sftp-cipher-aes128-ctr.ini

  # ssh transfer cipher
  cipher=aes128-ctr

Description 

• The global SystemProperty file will be applicable as described in the previous section YADE System Property Configuration. 
• In the following YADE Settings example two configuration files are created, both of them has configuration properties for the SFTP data provider;
  1. sftp-zlip-compression.ini :  effects packet compression during transfer over SFTP.
  2. sftp-cipher-aes128-ctr.ini : effects Cipher used by SFTP data provider.

• The sftp-zlip-compression.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos e.g. CopyLocal2SFTPUKSOS

• Similarly the sftp-cipher-aes128-ctr.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using configuration_files option, thus the Cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos e.g. CopySFTPJapanSOS2Local.

YADE Settings

• YADE Settings with system_property_files and configuration_files yade_settings.ini Expand source

  [globals]
  history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
  system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_properties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_properties.ini
   
  [protocol_fragment_sftp@sftp-uk.sos]
  protocol                            = sftp
  host                                = uk.sos
  port                                = 22
  user                                = london
  ssh_auth_method                     = publickey
  ssh_auth_file                       = ${HOME}/.ssh/id_rsa
  configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp-zlip-compression.ini
   
  [protocol_fragment_sftp@sftp-japan.sos]
  protocol                            = sftp
  host                                = japan.sos
  user                                = tokyo
  ssh_auth_method                     = publickey
  ssh_auth_file                       = ${HOME}/.ssh/id_rsa
  configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp-cipher-aes128-ctr.ini
  
  [CopyLocal2SFTPUKSOS]
  operation                           = copy
  file_spec                           = TEST-M-1111.TXT
  source_protocol                     = local
  source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
  target_include                      = protocol_fragment_sftp@sftp-uk.sos
  target_dir                          = /home/london/data/to_london/
   
  [CopySFTPJapanSOS2Local]
  operation                           = copy
  file_spec                           = TEST-M-2222.TXT
  source_protocol                     = sftp
  source_include                      = protocol_fragment_sftp@sftp-japan.sos
  source_dir                          = /home/tokyo/data/from_tokyo/
  target_protocol                     = local
  target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

References

Change Management References