Page History

Status

colour	Blue
title	Under Review

Table of Contents

Scope

YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba JCifs, SFTP protocol implemented by JSch library.
Data providers can make use of individual Java properties, e.g.
- for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer
or
- for SFTP to specify packet compression to zlib, compression level 9 or
Cipher
- specific ciphers such as blowfish-cbc,
aes192
- aes128-ctr etc.
YADE allows to specify Java property System Property files at a global level and individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-400
- Properties from Java property files properties are applied as system properties System Properties by YADE.
YADE allows to specify Java Application Property files as configuration files that are applied individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-403
- An application configuration file contains settings that are specific to an app/Java properties are applied as Application Properties from configuration files. Such properties are specific for a data provider.

YADE Java System Property

...

Files

YADE Configuration by System Property

...

Files

The Java properties configured using the system_property_files options are global properties available to all protocol fragments. The system_property_files option can be assign assigned a semicolon separated list of property files. A property file can contain on one or more properties as key value pairs.

Example

Sample System Properties

Following are two examples of such system property files , which contains that contain global Java properties for a data providersprovider. One system property file can also contain properties for multiple data providerproviders. At the run-time the YADE will automatically apply applicable properties to the respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

sftp_system_properties.ini
Code Block
language bash
title SystemProperyFile System Propery File sftp_system_proprerties.in
# SFTP properties known_hosts=/home/gollum.sos/.ssh/known_hosts remote_exec=true
smb_system_properties.ini
Code Block
language bash
title SystemProperyFile System Propery File smb_system_properties.ini
# SMB properties jcifs.smb.client.useExtendedSecurity=false

...

Sample YADE Configuration

Description

...

Globals section
Protocol Fragments section
Transfer Profiles section

...

Code Block

language	bash
title	YADE Settings file with system_property_files yade_settings.

...

ini
collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_properties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_properties.ini

[protocol_fragment_sftp@sftp-

...

YADE Settings

Code Block

language	bash
title	YADE Settings file with system_property_files yade_settings.ini
collapse	true

[globals]
historyuk.sos]
protocol                            = sftp
host       = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files                         = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_properties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_properties.ini

[protocol_fragment_sftp@sftp-uk.sos]
protocoluk.sos
port                                = sftp22
hostuser                                = uk.sos
portlondon
ssh_auth_method                       = publickey
ssh_auth_file         = 22
user             = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                  = london
ssh_auth_method         = sftp
host           = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-= japan.sos]
protocoluser                            = sftp
host           = tokyo
ssh_auth_method                     = japan.sos
userpublickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation        = tokyo
ssh_auth_method                     = publickeycopy
sshfile_auth_filespec                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation   = TEST-M-1111.TXT
source_protocol                       = copylocal
filesource_specdir                           = TEST-M-1111.TXT
source_protocol= /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = local
sourceprotocol_fragment_sftp@sftp-uk.sos
target_dir                          = /mnthome/r2d2/nobackuplondon/data/from_galadriel
target_includeto_london/

[CopySFTPJapanSOS2Local]
operation                           = protocol_fragment_sftp@sftp-uk.sos
target_dircopy
file_spec                           = /home/london/data/to_london/

[CopySFTPJapanSOS2Local]
operation  TEST-M-2222.TXT
source_protocol                         = copysftp
file_spec    source_include                       = TEST-M-2222.TXTprotocol_fragment_sftp@sftp-japan.sos
source_protocol   dir                  = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir     = /home/tokyo/data/from_tokyo/
target_protocol                     = /home/tokyo/data/from_tokyo/local
target_protocoldir                     = local
target_dir                          = = /mnt/r2d2/nobackup/data/to_galadriel

Explanations

YADE Configuration Files Configuration

The protocol fragment specific Java properties can be configured using configuration_files option on protocol fragment level . The configuration_files option can be assign a semicolon separated list of property files. A property file can contain on or more properties as key value pairs.

Example

Code Block

language	bash
title	SFTP packet compression configuration sftp-zlip-compression.ini

# ssh packet compression
compression.s2c=zlib@openssh.com,zlib,none
compression.c2s=zlib@openssh.com,zlib,none
compression_level=9

Code Block
language bash
title SFTP Cipher configuration sftp-cipher-aes128-ctr.ini
# ssh transfer cipher cipher=aes128-ctr

Description

The YADE settings consist of the following types of sections:
1. Globals section
2. Protocol Fragment sections
3. Transfer Profile sections
In the sample yade_settings.ini file, under the globals section the system property files are configure as semicolon separated list of files by the option system_property_files.
The properties defined in the files sftp_system_properties.ini and smb_system_properties.ini will be available to all protocol fragments.
The known_hosts and remote_exec system property will be available and applied to all the fragments using protocol SFTP, e.g. to the protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Java Application Property Files

YADE Confguration by Application Property Files

The protocol fragment specific Java properties can be configured using the configuration_files option at protocol fragment level. The configuration_files option can be assigned a semicolon separated list of property files. A property file can contain one or more properties as key value pairs.

Example

Sample Application Properties for Data Provider

Code Block

language	bash
title	SFTP packet compression configuration sftp-zlip-compression.ini

# ssh packet compression
compression.s2c=zlib@openssh.com,zlib,none
compression.c2s=zlib@openssh.com,zlib,none
compression_level=9

Code Block
language bash
title SFTP Cipher configuration
The global SystemProperty file will be applicable as described in the previous section YADE System Property Configuration.
In the following YADE Settings example two configuration files are created, both of them has configuration properties for the SFTP data provider;
sftp-zlip-compression.ini : effects packet compression during transfer over SFTP.

sftp-cipher-aes128-ctr.ini

: effects Cipher used by SFTP data provider.

The sftp-zlip-compression.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos e.g. CopyLocal2SFTPUKSOS
Similarly the sftp-cipher-aes128-ctr.ini is referred by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using configuration_files option, thus the Cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos e.g. CopySFTPJapanSOS2Local.

YADE Settings

...

language	bash
title	YADE Settings with system_property_files and configuration_files yade_settings.ini
collapse	true

...

# ssh transfer cipher cipher=aes128-ctr

Sample YADE Configuration

Code Block

language	bash
title	YADE Settings with system_property_files and configuration_files yade_settings.ini
collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/sftp_system_properties.ini ; /home/test/sos-berlin.com/jade_client/configurations/smb_system_properties.ini
 
[protocol_fragment_sftp@sftp-uk.sos]
protocol

...

sftp
host

...

          = uk.sos
port                                =

...

user                                =

...

london
ssh_auth_method                     = publickey
ssh_auth_file

...

            = ${HOME}/.ssh/id_rsa
configuration_files

...

= /home/test/sos-berlin.com/jade_client/configurations/sftp-zlip-compression.ini
 
[protocol_fragment_sftp@sftp-japan.sos]
protocol

...

     = sftp
host

...

japan.sos

...

user

...

  = tokyo
ssh_auth_method

...

publickey
ssh_auth_file

...

= ${HOME}/.ssh/id_rsa
configuration_files

...

= /home/test/sos-berlin.com/jade_client/configurations/sftp-cipher-aes128-ctr.ini

[CopyLocal2SFTPUKSOS]
operation                           =

...

copy
file_spec

...

          = TEST-M-1111.TXT
source_protocol                     =

...

local

...

source_

...

dir

...

= /mnt/r2d2/nobackup/data/from_galadriel
target_include                      =

...

protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /

...

home/

...

london/

...

data/

...

to_london/
 
[CopySFTPJapanSOS2Local]
operation                           =

...

copy
file_spec                           =

...

TEST-M-2222.TXT
source_protocol                     = sftp
source_include

...

                = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

Explanations

The global system property file will be applicable as explained in the previous chapter on YADE Java System Property Files.
In the YADE Settings sample two configuration files are created, both of them include configuration properties for the SFTP data provider;
1. sftp-zlip-compression.ini : effects packet compression during transfer by SFTP data provider.
2. sftp-cipher-aes128-ctr.ini : effects Cipher used by SFTP data provider.
= TEST-M-2222.TXT source_protocol = sftp source_include =
The sftp-zlip-compression.ini is referred to by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using the configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment
protocol_fragment_sftp@sftp-
japan.sos source_dir = /home/tokyo/data/from_tokyo/ target_protocol = local target_dir = /mnt/r2d2/nobackup/data/to_galadriel
uk.sos, e.g. CopyLocal2SFTPUKSOS.
Similarly the sftp-cipher-aes128-ctr.ini is referred to by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using the configuration_files option, thus the cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos, e.g. CopySFTPJapanSOS2Local.
At run-time the YADE will automatically apply applicable properties to respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

References

Change Management References

...

Space shortcuts

Page tree

Versions Compared

Old Version 12

New Version 13

Key

Scope

YADE Java System Property

Files

YADE Configuration by System Property

Files

Example

Sample System Properties

Sample YADE Configuration

Description

YADE Settings

Explanations

YADE Configuration Files Configuration

Example

Description

YADE Java Application Property Files

YADE Confguration by Application Property Files

Example

Sample Application Properties for Data Provider

YADE Settings

Sample YADE Configuration

Explanations

References

Change Management References