Product Knowledge Base

Space shortcuts

Page tree

Server-to-Server File Transfer (S2S) is a unique key feature of YADE. It provides the possibility to set-up a number of file transfers using a single server without installing clients/agents on each source/target host.

YADE Implementation Architecture

YADE Architecture Components

YADE includes the following main components:

  1. CLI (Command Line Interface) executed at shell level
  2. User Application Programming Interface (API)
  3. JITLJobs scheduling component of YADE
  4. YADE Engine
  5. Options Engine
  6. YADE Engine Interface
  7. VFS (Virtual File System) Interface (Data Provider)
    • FTP
    • SFTP
    • FTPS
    • HTTP/HTTPS FEATURE AVAILABILITY STARTING FROM RELEASE 1.7
    • WebDAV  FEATURE AVAILABILITY STARTING FROM RELEASE 1.8
    • Local (native copy)

YADE Server-to-Server File Transfer

YADE can transfer data using an intermediate server from a Source Server-to-Target Server (S2S) without touchdown on an intermediate server.

 

S2S implementation details:

  1. YADE uses "Streams" (or in common terminology: pipes) to transfer data from source to target.
  2. YADE opens an input stream from the source server and and output stream to the target server.
  3. YADE can use different data provider as source and target for S2S operation.
  4. By default YADE will transfer 32Kb chunks from input to output stream. You can increase the buffer size by setting the buffer_size parameter, but a larger buffer size may not result in higher transfer speed.
  5. YADE can process the content of the buffer between source and target transfer. By doing in-memory operations, even for multistage processing, YADE dramatically improves the transfer speed.
    1. Calculate Hashes
    2. Reserved for further development
      1. Filtering
      2. Encoding/Decoding
      3. Encryption/Decryption
  6. YADE transfers data from the source server to the target server without touchdown meaning that during transfer the data is never written to the file system on the intermediate server where YADE is running. YADE's key feature is in-memory transfer. During transfer YADE does not copy any data to the memory, but only chunks whose size is defined with the buffer_size parameter.

For more information about YADE Server-to-Server File Transfer configuration see Example for YADE Server-to-Server File Transfer.

YADE provides a generic implementation for Virtual File Systems or Data Providers that makes it easy to implement new protocols.