Hybrid Cloud Managed File Transfer Solution

Stonebranch helps you securely transfer files across on-prem, cloud, hybrid environments, and B2B partners. In this video, we will focus on cloud data transfer, a universal task that moves data directly between cloud storage services or file systems. It's fast, secure, and avoids temporary storage. To maximize performance, Cloud data transfer supports chunking of large files and parallel transfers when you're moving multiple files at once. It supports major platforms like AWS, Azure, Google Cloud, OneDrive, and now Microsoft OneLake for seamless Microsoft Fabric integration. Configuring the task is extremely simple. First, select the action you want to perform. For example, copy, move, or synchronize data between storage buckets. Once that's done, just choose your source and target system. In this demo, I'm streaming files from an Azure Data Lake Gen 2 bucket to an AWS S3 bucket. Next, you select which files you want to transfer. Cloud data transfer includes powerful filtering options. You can use a match pattern or regular expressions to select files and even control which folder levels are included. You can also define file name transformations. For example, adding prefixes or suffixes, or applying complex renaming rules using regex. In my example, I'm adding a date suffix and capitalizing the file names. Before running the transfer, I can perform a dry run. This simulates the entire transfer process using the current network transfer rate, giving me a realistic forecast of how long the transfer will take. Now let's run the transfer. As you can see, four files have been transferred successfully, and their names have been automatically updated to the date and the capitalization rules applied. The file transfer runs as a task within the Universal Controller, just like any other task. That means you can start it on a calendar based using a time trigger, when a file arrives, using a file monitor, after another task completes with a task monitor, from a message in a queue like Kafka or AWS SQS, or in real time triggered by a webhook event. For example, when a file is uploaded to an S3 bucket. The webhook option is especially powerful because it removes the need to constantly poll for new files. It reacts instantly. You can also integrate the cloud data transfer into any workflow. For example, in an ETL process, you can fetch data from any cloud source and trigger an ETL process as soon as the data is available. Here's one example. When a file is uploaded to a SharePoint folder, a data stream is triggered automatically from a SharePoint folder to an AWS S3 bucket. Once the data reaches S3, the AWS Glue ETL task runs automatically, followed by a Power BI refresh task. Let us make it a bit more complex by adding a B2B file transfer and a mainframe to Kubernetes transfer. Let's extend the workflow now. I will add a direct transfer from a mainframe to OpenShift Kubernetes. This is a new feature of UDM 7.9, which is now supporting OpenShift routes. I add a Kubernetes job task that does calculations based on the data from the mainframe, then transfer the result from OpenShift to AWS S3. Next, I use UDM Data Mover Gateway to collect files from SFTP server using mget and upload them to an S3 bucket. Once all files are available, I run the AWS Glue ETL test, then refresh Power BI, and finally use another Universal Data Mover gateway task to upload the process files where I input to an SFTP server. Let's launch the workflow now. I will now upload the sales data to my SharePoint folder. As soon as the data is uploaded, the workflow starts and the different transfers are executed. Monitoring your transfer is easy and flexible depending on your role. Operations teams can monitor progress through the Universal Controller activity monitor, the real-time workflow view, or a customized dashboard. Business users can monitor via the Universal Portal on their phone. By selecting the Cloud Data Transfer Business Service, I can see only the tasks relevant for this use case. Finally, observability teams can monitor all activities through Grafana or Dynatrace. Shown here is a Grafana dashboard that provides OpenTelemetry-based observability for all data transfers. To summarize, with Universal Automation Center, you can stream data between any cloud storage system, transfer data from mainframes or servers into Kubernetes, move data from Kubernetes to any cloud provider, exchange files with B2B partners, and combine all of these into a single automated workflow. All of this can be monitored and managed through the Universal Controller web interface, the Universal self-service Portal, or via OpenTelemetry-based observability, in example given Grafana and Dynatrace.