How to configure Java property files for data provider

IN PROGRESS

Scope

• YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba.
• Data providers can make use of individual Java properties, e.g. for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer or for SFTP to specify packet compression to zlib, compression level 9 or Cipher blowfish-cbc,aes192-ctr.
• YADE allows to specify Java property files at a global level and individually per transfer fragment.
  • YADE-400 - Getting issue details... STATUS
  • Properties from Java property files are applied as system properties by YADE.
• YADE allows to specify configuration files individually per transfer fragment.
  • YADE-403 - Getting issue details... STATUS
  • An application configuration file contains settings that are specific to an app/data provider.
    
    

YADE Java Property File Configuration

YADE System Property Configuration

The Java properties configured using  System Property Files are global properties available to all protocol fragments. The SystemPropertyFiles option can be assign a semicolon separated list of property files. The property files can contain on or more properties as key value pairs.

Following is the example of such a System Property File which contains global properties for multiple data providers. The YADE will apply applicable properties to respective data provider e.g. SFTP properties to JSch data provider and SMB properties to JCifs data provider. 

• sftp_system_proprieties.ini

  SystemProperyFile sftp_system_proprieties.in

  # SFTP properties
  known_hosts=/home/gollum.sos/.ssh/known_hosts
  remote_exec=true

• smb_system_proprieties.ini

  SystemProperyFile smb_system_proprieties.ini

  # SMB properties
  jcifs.smb.client.useExtendedSecurity=false

Example

Description 

• The YADE profile has three sections 
  1. Globals section 
  2. ProtocolsFragments section
  3. Transfer Profiles section
• The system properties files configure under globals section with option system_property_files will be available to all the protocol fragments. At the run time the YADE will automatically apply applicable properties to respective data provider e.g. SFTP properties to JSch data provider and SMB properties to JCifs data provider.

• The known_hosts and remote_exec system property will be available and applied to all the fragments using proticol SFTP  e.g. to protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Settings

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_proprieties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_proprieties.ini

[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host                                = uk.sos
port                                = 22
user                                = london
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host                                = japan.sos
user                                = tokyo
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec                           = TEST-M-1111.TXT
source_protocol                     = local
source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation                           = copy
file_spec                           = TEST-M-2222.TXT
source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

YADE Configuration Files Configuration

The ConfigurationFiles are properties available only to the protocol fragments they are assigned. The ConfigurationFile option can be assign a semicolon separated list of property files. The property files can contain on or more properties as key value pairs.

Example

• SFTP packet compression configuration sftp-zlip-compression.ini

  compression.s2c=zlib@openssh.com,zlib,none
  compression.c2s=zlib@openssh.com,zlib,none
  compression_level=9
  

• SFTP Cipher configuration sftp-cipher-aes128-ctr.ini

  cipher=aes128-ctr

Description 

• The global SystemProperty file will be applicable as described in the previous section YADE System Property Configuration. 
• In the following example two configuration files are created, both of them has configuration properties for the SFTP data provider;
  1. sftp-zlip-compression.ini :  effects packet compression during transfer over SFTP.
  2. sftp-cipher-aes128-ctr.ini : effects Cipher used by SFTP data provider.

• The sftp-zlip-compression.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos e.g. CopyLocal2SFTPUKSOS

• Similarly the sftp-cipher-aes128-ctr.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using configuration_files option, thus the Cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos e.g. CopySFTPJapanSOS2Local.

YADE Settings

• YADE Settings system_property_files Expand source

  [globals]
  history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
  system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_proprieties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_proprieties.ini
   
  [protocol_fragment_sftp@sftp-uk.sos]
  protocol                            = sftp
  host                                = uk.sos
  port                                = 22
  user                                = london
  ssh_auth_method                     = publickey
  ssh_auth_file                       = ${HOME}/.ssh/id_rsa
  configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp-zlip-compression.ini
   
  [protocol_fragment_sftp@sftp-japan.sos]
  protocol                            = sftp
  host                                = japan.sos
  user                                = tokyo
  ssh_auth_method                     = publickey
  ssh_auth_file                       = ${HOME}/.ssh/id_rsa
  configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp-cipher-aes128-ctr.ini
  
  [CopyLocal2SFTPUKSOS]
  operation                           = copy
  file_spec                           = TEST-M-1111.TXT
  source_protocol                     = local
  source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
  target_include                      = protocol_fragment_sftp@sftp-uk.sos
  target_dir                          = /home/london/data/to_london/
   
  [CopySFTPJapanSOS2Local]
  operation                           = copy
  file_spec                           = TEST-M-2222.TXT
  source_protocol                     = sftp
  source_include                      = protocol_fragment_sftp@sftp-japan.sos
  source_dir                          = /home/tokyo/data/from_tokyo/
  target_protocol                     = local
  target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

References

Change Management References