Page History

Status

colour	Blue
title	Under Review

Table of Contents

Scope

YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba JCifs, SFTP protocol implemented by JSch library.
Data providers can make use of individual Java properties, e.g.
- for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer
or
- for SFTP to specify packet compression to zlib, compression level 9 or
Cipher
- specific ciphers such as blowfish-cbc,
aes192
- aes128-ctr etc.
YADE allows to specify Java property System Property files at a global level and individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-400
- Properties from Java property files properties are applied as system properties System Properties by YADE.
YADE allows to specify Java Application Property files as configuration files that are applied individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-403
- An application configuration file contains settings that are specific to an app/Java properties are applied as Application Properties from configuration files. Such properties are specific for a data provider.

YADE Java System Property

...

Files

YADE Configuration by System Property

...

Files

The Java properties configured using the system_property_files options are global properties available to all protocol fragments. The system_property_files option can be assign assigned a semicolon separated list of property files. A property file can contain on one or more properties as key value pairs.

Example

Sample System Properties

Following are two examples of such system property files , which contains that contain global Java properties for a data providersprovider. One system property file can also contain properties for multiple data providerproviders. At the run-time the YADE will automatically apply applicable properties to the respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

sftp_system_.properties.ini
Code Block
language bash
title SystemProperyFile System Propery File sftp_system_.proprerties.in
# SFTP properties known_hosts=/home/gollum.sos/.ssh/known_hosts remote_exec=true
smb_system_.properties.ini
Code Block
language bash
title SystemProperyFile System Propery File smb_system_.properties.ini
# SMB properties jcifs.smb.client.useExtendedSecurity=false

...

Sample YADE Configuration

Description

...

Globals section
Protocol Fragments section
Transfer Profiles section

...

Code Block

language	bash
title	YADE Settings file with system_property_files yade_settings.ini

...


collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties

[protocol_fragment_sftp@sftp-uk.sos

...

YADE Settings

Code Block

language	bash
title	YADE Settings file with system_property_files yade_settings.ini
collapse	true

[globals]
history]
protocol                            = sftp
host     = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files                           = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_properties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_properties.ini

[protocol_fragment_sftp@sftp-uk.sos]
protocoluk.sos
port                                 = sftp22
hostuser                                = uk.sos
portlondon
ssh_auth_method                     = publickey
ssh_auth_file          = 22
user            = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                   = london
ssh_auth_method        = sftp
host            = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocoluser                                = sftp
hosttokyo
ssh_auth_method                     = publickey
ssh_auth_file           = japan.sos
user           = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                    = tokyo
ssh_auth_method               = copy
file_spec      = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operationTEST-M-1111.TXT
source_protocol                     = local
source_dir      = copy
file_spec                   = /mnt/r2d2/nobackup/data/from_galadriel
target_include       = TEST-M-1111.TXT
source_protocol                     = local
sourceprotocol_fragment_sftp@sftp-uk.sos
target_dir                          = /mnthome/r2d2london/nobackup/data/from_galadriel
target_includeto_london/

[CopySFTPJapanSOS2Local]
operation                           = protocol_fragment_sftp@sftp-uk.sos
target_dircopy
file_spec                           = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation     TEST-M-2222.TXT
source_protocol                      = copysftp
filesource_specinclude                           = TEST-M-2222.TXT
source_protocol= protocol_fragment_sftp@sftp-japan.sos
source_dir                          = sftp
source_include /home/tokyo/data/from_tokyo/
target_protocol                     = protocol_fragment_sftp@sftp-japan.sos
sourcelocal
target_dir                          = /mnt/homer2d2/tokyonobackup/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

YADE Configuration Files Configuration

The protocol fragment specific Java properties can be configured using configuration_files option on protocol fragment level . The configuration_files option can be assign a semicolon separated list of property files. A property file can contain on or more properties as key value pairs.

Example

...

language	bash
title	SFTP packet compression configuration sftp-zlip-compression.ini

...

to_galadriel

Explanations

The YADE settings consist of the following types of sections:
1. Globals section
2. Protocol Fragment sections
3. Transfer Profile sections
In the sample yade_settings.ini file, under the globals section the system property files are configure as semicolon separated list of files by the option system_property_files.
The properties defined in the files sftp_system.properties and smb_system.properties will be available to all protocol fragments.
The known_hosts and remote_exec system property will be available and applied to all the fragments using protocol SFTP, e.g. to the protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Java Application Property Files

YADE Confguration by Application Property Files

The protocol fragment specific Java properties can be configured using the configuration_files option at protocol fragment level. The configuration_files option can be assigned a semicolon separated list of property files. A property file can contain one or more properties as key value pairs.

Example

Sample Application Properties for Data Provider

Code Block

language	bash
title	SFTP Cipher packet compression configuration sftp-cipher-aes128-ctr.ini_zlip_compression.properties

# ssh transferpacket ciphercompression
cipher=compression.s2c=zlib@openssh.com,zlib,none
compression.c2s=zlib@openssh.com,zlib,none
compression_level=9

Code Block
language bash
title SFTP Cipher configuration sftp_cipher_aes128-ctr

Description

The global SystemProperty file will be applicable as described in the previous section YADE System Property Configuration.
In the following YADE Settings example two configuration files are created, both of them has configuration properties for the SFTP data provider;
1. sftp-zlip-compression.ini : effects packet compression during transfer over SFTP.
2. sftp-cipher-aes128-ctr.ini : effects Cipher used by SFTP data provider.
The sftp-zlip-compression.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos e.g. CopyLocal2SFTPUKSOS
Similarly the sftp-cipher-aes128-ctr.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using configuration_files option, thus the Cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos e.g. CopySFTPJapanSOS2Local.

YADE Settings

...

language	bash
title	YADE Settings with system_property_files and configuration_files yade_settings.ini
collapse	true

.properties
# ssh transfer cipher cipher=aes256-ctr

Sample YADE Configuration

Code Block

language	bash
title	YADE Settings with system_property_files and configuration_files yade_settings.ini
collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties
 
[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host

...

uk.sos
port

...

           = 22
user

...

          = london
ssh_auth_method                     =

...

publickey
ssh_auth_file

...

= ${HOME}/.ssh/id_rsa
configuration_files

...

= /home/test/sos-berlin.com/jade_client/configurations/sftp_zlib_compression.properties
 
[protocol_fragment_sftp@sftp-japan.sos]
protocol                            =

...

sftp
host

...

       = japan.sos
user                                =

...

tokyo
ssh_auth_method                     = publickey
ssh_auth_file

...

              = ${HOME}/.ssh/id_rsa
configuration_files                 =

...

/home/test/sos-berlin.com/jade_client/configurations/sftp_cipher_aes128-ctr.properties

[CopyLocal2SFTPUKSOS]
operation                           =

...

copy

...

file_spec

...

      = TEST-M-1111.TXT
source_protocol                     =

...

local
source_dir                          = /

...

mnt/r2d2/

...

nobackup/data/from_galadriel
target_include                      =

...

protocol_fragment_sftp@sftp-uk.sos
target_dir                          =

...

/home/london/data/to_london/
 
[CopySFTPJapanSOS2Local]
operation                           =

...

copy

...

file_

...

spec                           =

...

TEST-M-2222.TXT
source_protocol

...

sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos

...

source_dir                          = /home/

...

tokyo/data/

...

from_

...

tokyo/

...

target_protocol                     = local
target_dir

...

                  = /mnt/r2d2/nobackup/data/to_galadriel

Explanations

The global system property file will be applicable as explained in the previous chapter on YADE Java System Property Files.
In the YADE Settings sample two configuration files are created, both of them include configuration properties for the SFTP data provider;
1. sftp_zlip_compression.properties : effects packet compression during transfer by SFTP data provider.
2. sftp_cipher_aes128-ctr.properties : effects Cipher used by SFTP data provider.
The sftp_zlip_compression.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using the configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos, e.g. CopyLocal2SFTPUKSOS.
Similarly the sftp_cipher_aes128-ctr.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using the configuration_files option, thus the cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos, e.g. CopySFTPJapanSOS2Local.
At run-time the YADE will automatically apply applicable properties to respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.
= TEST-M-2222.TXT source_protocol = sftp source_include = protocol_fragment_sftp@sftp-japan.sos source_dir = /home/tokyo/data/from_tokyo/ target_protocol = local target_dir = /mnt/r2d2/nobackup/data/to_galadriel

References

Change Management References

...

Space shortcuts

Page tree

Versions Compared

Old Version 12

New Version Current

Key

Scope

YADE Java System Property

Files

YADE Configuration by System Property

Files

Example

Sample System Properties

Sample YADE Configuration

Description

YADE Settings

YADE Configuration Files Configuration

Example

Explanations

YADE Java Application Property Files

YADE Confguration by Application Property Files

Example

Sample Application Properties for Data Provider

Description

YADE Settings

Sample YADE Configuration

Explanations

References

Change Management References