Scope

• YADE should support file transfer operations with Azure Blob Storage.
• Technically Azure file transfer operations preferably make use of the HTTPS protocol and a number of query parameters and headers. The implementation with YADE therefore is fairly straightforward.
• The sticking point is about what authentication methods should be supported and how they would be integrated:
  • If blob containers are made publicly available and require no authentication then the YADE supports this out-of-the-box.
  • If Azure authentication methods are applied then different implementation strategies are to be considered. Azure supports a range of authentication methods - we picked two of them for this proposal:
    • Use of Shared Key authentication, see Authorize with Shared Key
    • Use of Shared Access Signature authentication (SAS), see Create an Account SAS
• There are impacts on job operation depending on the authentication method that is preferred by an organization.

Authentication Methods

Find a preliminary comparison of authentication methods:

• From the above comparison Shared Access Signatures are superior concerning more fine-grained access to resources.
• At the same time Shared Key authentication suggests ease of use as a single Shared Key can be used to authenticate any file transfer operations for an unlimited duration.
• Technically a single Shared Access Signature can be applied in a similar way allowing any file transfer operation on any blobs in a given container for a longer period, e.g. the next 100 years.

Operation

• Considering use of authentication methods with file transfer jobs there is a final statement from SOS: implementation of any authentication method is out of scope of the JobScheduler product. 
  • Implementation of authentication methods is within the responsibility of the organization that uses JobScheduler. Reasons for this include that
    • users should not trust any 3rd party implementation of authentication methods,
    • a number of authentication methods are offered by Azure that allow an individual choice according to security requirements of an organization,
    • there is ongoing development with Azure about improvement of authentication methods.
  • Implementation of file transfer operations for Azure blob storage is within the scope of the YADE.
• Technically this offers the following options:
  • An organization can create a single Shared Key or Shared Access Signature that represents a constant value that is made available to all YADE file transfer jobs.
  • An organization can create individual Shared Keys or Shared Access Signatures that are used with groups of YADE file transfer jobs or with individual jobs.

Implementation

User Tasks: Manage Shared Keys and Shared Access Signatures

Creating Shared Keys and Shared Access Signatures

• Find examples for creating Shared Keys and SAS tokens on the fly and for using Azure authentication methods and respective file transfer operations from the following articles:
  • Feature Proposal - Azure Blob Storage - Shared Key Authentication
  • Feature Proposal - Azure Blob Storage - Shared Access Signatures
• The examples are far from being perfect, however, they might be useful as a basis for a user's implementation.

SOS Tasks: YADE Integration

Implementation Strategies

• When using a single Shared Key or Shared Access Signature for all YADE file transfer jobs then
  • this could be added as a constant value to the JobScheduler configuration, e.g. with a parameter to the ./config/scheduler.xml file.
  • this could be added to a Credential Store that would be accessed by any YADE file transfer jobs.
• When using individual Shared Keys or Shared Access Signatures per YADE file transfer job then a Monitor should be implemented that is assigned on a per job basis and that can be parameterized to use the key or signature specified by a job parameter or order parameter.
• Should Shared Keys or Shared Access Signatures have to be renewed on a regular basis then this can be performed by a job, e.g. running daily, that is implemented by a user of JobScheduler.

YADE Changes

• Accept for both Shared Keys and Shared Access Signatures the respective configuration items:
  • For both authentication methods
    • Add to URL
      • Resources
      • container resource
      • blob resource
      • Example
        • https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>
        • https://yade.blob.core.windows.net/yade/test.txt
    • Add HTTPS headers:
      • x-ms-blob-type : BlockBlob
      • x-ms-date : <date of request>
      • x-ms-version : <version as specified by user>
  • Shared Key
    • Add HTTPS authorization header:
      • Authorization : SharedKey <requester account specified by user>:<signature specified by user>`
  • Shared Access Signatures
    • Add query parameters to URL for operation to list blobs in a container
      • https://<owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<query parameters>
      • https://yade.blob.core.windows.net/yade/test.txt?restype=container&comp=list
    • Add SAS token to URL:
      • https://<blob storage owner account specified by user>.blob.core.windows.net/<container specified by user>/<blob specified by user>?<SAS token specified by user>
• Adjust the YADE XSD Schema to allow configuration of any number of HTTP headers
  • at a global level 
  • per profile
  • on a per job basis by job/order parameters and when using the YADE standalone client by command line parameters.
•  Delimitation
  • The implementation is about integrating the above configuration items into HTTPS headers and query parameters.
  • The implementation is not about developing a new data provider that is subject to a separate feature request.

References

YADE-561 - Getting issue details... STATUS

Resources