Cloud Data Transfer

This Universal Extension provides the capability to perform data transfers between cloud based storage services, as well as local or distributed file systems. Transfers are fast and secure since data are streamed from one storage to another with no intermediate storage taking place.

Multiple storage systems are supported (an overview can be found here). Integrations within this solution package include:

• Amazon S3

• Google Cloud Storage

• Microsoft OneDrive Business, including Sharepoint

• Microsoft Azure Blob Storage

• Hadoop Distributed File Storage (HDFS)

• Local file system (Linux, Windows)

• HTTP(S) URL

Key Features

This Universal Extension supports the following key features:

Actions

• Copy, move, synchronize data between two storages. 

• Copy a URL's content and to cloud or local destination without saving it in temporary storage. 

• List data on a storage, including listing with details or in JSON format for machine parsing. 

• Create objects on a storage. 

• Delete objects from a storage.

Features

• Fast transfers for objects stored in the same region.

• Preserves always timestamps and verifies checksums.

• Supports encryption, caching, compression, chunking.

• Dynamic token updates for OneDrive Business cloud storage, observing the OneDrive business refresh token flow.

• Support for dry runs. Allows users to execute a Universal Task without making any permanent changes on the target storage.

• Advanced filtering capability for files or objects to be listed or transferred.

• Option to mark the Universal Task as Failed when no files have been transferred.

• List of overwrite options for existing data.

• Additional customized options.

• Observability capabilities, providing users with detailed statistical insights for data transfers across all task instances.

Output

• Progress of the selected Action is visible, during Universal Task Instance execution.

• Text or JSON formatted output.

Premium Features

• Monitor files inside OneLake Lakehouses.

To receive Cloud Data Transfer Premium contact Stonebranch.

What's New 4.8.0:

Enhancements

• Fewer Universal Events published during long transfers, reducing the load on the Universal Controller. 
• Updated dependencies. 
• Security Improvements.

Bugfixes 

• Object names with consecutive, leading or trailing spaces are now listed correctly. 
• Fixed Actions failing on Windows Agents when the name of the Configuration File Script has no file extension (e.g. clouddt-conf instead of clouddt.conf).