- Data Pipeline Automation
Complex automation workflows can be a thing of beauty. This is especially true when they span multiple systems in a hybrid IT environment. In this article, we share some particularly cool workflows that will give you a sense of what’s possible with the Universal Automation Center (UAC).
For those unfamiliar with UAC, the first thing you need to know is that the platform plays in a category of solutions known as the Service Orchestration and Automation Platform, or SOAP for short. The SOAP category is an evolution from traditional workload automation, which essentially automates IT tasks for mainframe and distributed environments. SOAPs, on the other hand, help enterprises orchestrate automation workflows across on-prem, cloud and containerized microservices. Plus, SOAPs have built-in managed file transfer capabilities.
Stonebranch has worked with our customers to create some stunning workflows that solve specific problems. Read on to see visual examples pulled directly from UAC's visual drag-and-drop workflow editor.
Big Data Pipeline Workflow
This workflow illustrates how to orchestrate a big data pipeline. This workflow's primary goal is to replace custom scripts and point-to-point integrations with UAC's centralized scheduler. Managing all the automated processes from the UAC empowers the DataOps team with proactive alerts and the ability to quickly root-cause issues if something ever breaks down.
Within the workflow, the designer created a nice and clean process to orchestrate data movement through a traditional data pipeline, from source to ingestion to storage to delivery. Each tool used along the pipeline is connected directly with the UAC using highly secure integration standards.
A Quick Point About Integration
Before we go any further, let's talk about integrations. None of these workflows are possible without the ability to connect an IT automation solution, like a SOAP, to third-party platforms and applications via integration. Whether you're using the UAC or another automation platform, it's essential to focus your integration approach on security and encryption. You'll want to speak with your vendor about their method for both on-prem and cloud-based integration. Plus, be sure to ask about container-based integrations, which increase your ability to orchestrate more complex workflows.
This is a simplified representation of a UAC workflow used by a Stonebranch customer. By enabling a continuous delivery model, this DevOps workflow dramatically accelerates the launch of new commission models for the customer’s independent salesforce of more than 600,000 sellers. With UAC, new commission workflow models are now tested and promoted to production in a fraction of the time it used to take.
In actuality, the workflow has more than 400 steps; however, the simplified representation below gives a good idea of how the customer is using 100% event-driven workflows that are designed for testing before going into production. To learn more about how our customer, Vorwerk, accomplishes this, please check out this solution paper.
Data Transfer from On-Prem to Cloud
For many organizations moving to cloud-oriented hybrid IT environments, streamlined management and cost-efficiency are key objectives. Before reaching those goals, however, coordinating data via managed file transfer and ETL can be challenging.
Flexibility is one of UAC’s best attributes when it comes to setting up and running on-prem to cloud file transfer workflows. The workflow below is, in a sense, both a file-transfer and an ETL pipeline: the top half represents the extraction process, the bottom half is transformation, and at the very bottom all the data is loaded into a master database (the SQL Server SSIS box).
From a file-transfer perspective, on the left side of the workflow, we see files transferred to the cloud; a mini-DevOps workflow is represented at top right; and remote manual approval (the box with the green figure) sits in the middle. Of note, the designated approver does not need to go into the UAC user interface; they can hit the approve button from within, for example, an Microsoft Teams channel and the workflow will continue.
Once the workflow is approved, the analytical layer kicks off, where all steps are performed automatically. UAC can connect to any database, as illustrated by the SQL and Oracle databases in the bottom row.
Managed File Transfer: Mainframe to OpenShift Cluster
The three UAC workflows we’ve shared so far are pretty complex, but simplicity has its own appeal. Pound for pound, this short but sweet workflow is one of UAC’s strongest performers.
As container-based applications grow in popularity, on-prem platforms and applications need to connect to containerized applications in the cloud. That’s what we see represented in this workflow. One thing to note here: the workflow dynamically adjusts to the number of virtual machines, or pods, you’ve started in Kubernetes. So, while at the far right you’ll see two file transfers started, the number of files we could have shown is practically unlimited (well, except by the size of the page).
Automated Workflows Unlimited
If you made it to this point in the article, we hope you walk away genuinely inspired with new ideas about how you can create automated workflows for your business. Given the depth and breadth of UAC's capabilities, there are so many different ways to orchestrate automated IT processes.
Of course, in addition to automated IT processes, you can also automate business processes. For example, one customer designed a workflow that automates IT processes and business approval processes to help provision laptop computers for new hire on-boarding. But we'll save this one for another post.
We'd love to hear from you: Have an idea for a workflow you'd like to create? Or perhaps you've already made a cool workflow that you want to share with our audience? Please get in touch. We'd love to help.
Start Your Automation Initiative Now
Schedule a Live Demo with a Stonebranch Solution Expert