Page History

Status

colour	Yellow
title	In Progress

Table of Contents

Scope

YADE implements a Virtual File System that supports a number of data providers, e.g. SMB/CIFS protocol implementation by Samba JCifs, SFTP protocol implemented by JSch library.
Data providers can make use of individual Java properties, e.g.
- for SMB/CIFS to specify compatibility levels for authentication with NTLMv2 and newer
or
- for SFTP to specify packet compression to zlib, compression level 9 or
Cipher
- specific ciphers such as blowfish-cbc,
aes192
- aes128-ctr etc.
YADE allows to specify Java property System Property files at a global level and individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-400
- Properties from Java property files properties are applied as system properties System Properties by YADE.
YADE allows to specify Java Application Property files as configuration files that are applied individually per file transfer fragment.
- Jira
  server SOS JIRA
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 6dc67751-9d67-34cd-985b-194a8cdc9602
  key YADE-403
- An application configuration file contains settings that are specific to an app/Java properties are applied as Application Properties from configuration files. Such properties are specific for a data provider.

YADE Java System Property

...

Files

YADE Configuration by System Property

...

Files

The System property is a global property Java properties configured using the system_property_files options are global properties available to all protocol fragments and transfer profiles. The SystemPropertyFiles system_property_files option can be assign assigned a semicolon separated list of property files. The A property files file can contain on one or more properties as key value pairs.

...

Example

...

language	bash
title	SystemProperyFile hashed_known_hosts.ini

...

known_hosts=/home/gollum.sos/.ssh/known_hosts

Sample System Properties

Following are two examples of such system property files that contain global Java properties for a data provider. One system property file can also contain properties for multiple data providers. At run-time the YADE will automatically apply applicable properties to the respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

sftp_system.properties
Code Block
language bash
title System Propery File sftp_system.proprerties
# SFTP properties known_hosts=/home/gollum.sos/.ssh/known_hosts remote_exec=true
smb_system.properties
Code Block
language bash
title System Propery File smb_system.properties
# SMB properties jcifs.smb.client.useExtendedSecurity=false

Sample YADE Configuration

Code Block

language	bash
title	YADE Settings file with system_property_files yade_settings.ini

Example

Description

The YADE profile has three sections 1. globals section 2. Protocol sections and 3. Transfer profile sections.
The system properties configure under globals section with option system_property_files will be available to all the protocol fragments and transfer profiles.
In following example the location of hashed known hosts file is configured as name value pair in the file hashed_known_hosts.ini.
The known_hosts system property will be available and applied to all the fragments and profile using SFTP data provider e.g. to protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Settings

Code Block

language	text
title	YADE Settings system_property_files
collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/hashedsftp_known_hosts.ini

[system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties

[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host                                = uk.sos
port                                = 22
user                                = london
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host                                = japan.sos
user                                = tokyo
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec                           = TEST-M-1111.TXT
source_protocol                     = local
source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /home/london/data/to_london/

[CopySFTPJapanSOSCopySFTPJapanSOS2Local]
operation                           = copy
file_spec                           = TEST-M-2222.TXT
source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

...

Explanations

YADE Configuration Files Configuration

The ConfigurationFiles are properties available only to the protocol fragments they are assigned. The ConfigurationFile option can be assign a semicolon separated list of property files. The property files can contain on or more properties as key value pairs.

Example

Code Block

language	bash
title	SFTP packet compression configuration sftp-zlip-compression.ini

compression.s2c=zlib@openssh.com,zlib,none
compression.c2s=zlib@openssh.com,zlib,none
compression_level=9

Code Block
language bash
title SFTP Cipher configuration sftp-cipher-aes128-ctr.ini
cipher=aes128-ctr

Description

The global SystemProperty file will be used as described in the previous section.
In following example the two configuration files are created, both of them has configuration properties for the SFTP data provider
1. sftp-zlip-compression.ini : effects packet compression during transfer over SFTP.
2. sftp-cipher-aes128-ctr.ini : effects Cipher used by SFTP data provider.
The sftp-zlip-compression.ini is being assigned to the protocol fragment protocol_fragment_sftp@sftp-uk.sos using, thus the packet compression properties will only be applicable to the protocol fragment protocol_fragment_sftp@sftp-uk.sos.
Similarly the sftp-cipher-aes128-ctr.ini is being assigned to the protocol fragment protocol_fragment_sftp@sftp-japan.sos, thus the Cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos.

YADE Settings

...

The YADE settings consist of the following types of sections:
1. Globals section
2. Protocol Fragment sections
3. Transfer Profile sections
In the sample yade_settings.ini file, under the globals section the system property files are configure as semicolon separated list of files by the option system_property_files.
The properties defined in the files sftp_system.properties and smb_system.properties will be available to all protocol fragments.
The known_hosts and remote_exec system property will be available and applied to all the fragments using protocol SFTP, e.g. to the protocol fragments protocol_fragment_sftp@sftp-uk.sos and protocol_fragment_sftp@sftp-japan.sos.

YADE Java Application Property Files

YADE Confguration by Application Property Files

The protocol fragment specific Java properties can be configured using the configuration_files option at protocol fragment level. The configuration_files option can be assigned a semicolon separated list of property files. A property file can contain one or more properties as key value pairs.

Example

Sample Application Properties for Data Provider

Code Block

language	bash
title	SFTP packet compression configuration sftp_zlip_compression.properties

# ssh packet compression
compression.s2c=zlib@openssh.com,zlib,none
compression.c2s=zlib@openssh.com,zlib,none
compression_level=9

Code Block
language bash
title SFTP Cipher configuration sftp_cipher_aes128-ctr.properties
# ssh transfer cipher cipher=aes256-ctr

Sample YADE Configuration

Code Block

language	bash
title	YADE Settings with system_property_files and configuration_files yade_settings.ini

...


collapse	true

[globals]
history                             = /home/test/sos-berlin.com/jade_client/logs/jade_history.csv
system_property_files               = /home/test/sos-berlin.com/jade_client/configurations/

...

sftp_system.properties ; /home/test/sos-berlin.com/jade_client/configurations/smb_system.properties
 
[protocol_fragment_sftp@sftp-uk.sos]
protocol                            = sftp
host                                = uk.sos
port                                = 22
user                                = london
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa
configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp

...

_zlib_compression.

...

properties
 
[protocol_fragment_sftp@sftp-japan.sos]
protocol                            = sftp
host                                = japan.sos
user                                = tokyo
ssh_auth_method                     = publickey
ssh_auth_file                       = ${HOME}/.ssh/id_rsa
configuration_files                 = /home/test/sos-berlin.com/jade_client/configurations/sftp

...

_cipher

...

_aes128-ctr.

...

properties

[CopyLocal2SFTPUKSOS]
operation                           = copy
file_spec                           = TEST-M-1111.TXT
source_protocol                     = local
source_dir                          = /mnt/r2d2/nobackup/data/from_galadriel
target_include                      = protocol_fragment_sftp@sftp-uk.sos
target_dir                          = /home/london/data/to_london/

...

CopySFTPJapanSOS2Local]
operation                           = copy
file_spec                           = TEST-M-2222.TXT
source_protocol                     = sftp
source_include                      = protocol_fragment_sftp@sftp-japan.sos
source_dir                          = /home/tokyo/data/from_tokyo/
target_protocol                     = local
target_dir                          = /mnt/r2d2/nobackup/data/to_galadriel

Explanations

The global system property file will be applicable as explained in the previous chapter on YADE Java System Property Files.
In the YADE Settings sample two configuration files are created, both of them include configuration properties for the SFTP data provider;
1. sftp_zlip_compression.properties : effects packet compression during transfer by SFTP data provider.
2. sftp_cipher_aes128-ctr.properties : effects Cipher used by SFTP data provider.
The sftp_zlip_compression.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-uk.sos using the configuration_files option, thus the packet compression properties will only be applicable to the transfers using protocol fragment protocol_fragment_sftp@sftp-uk.sos, e.g. CopyLocal2SFTPUKSOS.
Similarly the sftp_cipher_aes128-ctr.properties is referred to by the protocol fragment protocol_fragment_sftp@sftp-japan.sos using the configuration_files option, thus the cipher aes128-ctr will only be applicable to the transfers using the protocol fragment protocol_fragment_sftp@sftp-japan.sos, e.g. CopySFTPJapanSOS2Local.
At run-time the YADE will automatically apply applicable properties to respective data provider, e.g. SFTP properties to the JSch data provider and SMB properties to the JCifs data provider.

References

Change Management References

...

Space shortcuts

Page tree

Versions Compared

Old Version 6

New Version Current

Key

Scope

YADE Java System Property

Files

YADE Configuration by System Property

Files

Example

Sample System Properties

Sample YADE Configuration

Example

Description

YADE Settings

Explanations

YADE Configuration Files Configuration

Example

Description

YADE Settings

YADE Java Application Property Files

YADE Confguration by Application Property Files

Example

Sample Application Properties for Data Provider

Sample YADE Configuration

Explanations

References

Change Management References