- Data Pipeline Automation
The first week of Stonebranch Online 2021 has come to a close, and with it, two engaging webinars on the state of the IT automation industry and how DataOps methodologies can help orchestrate data pipelines. Scroll down to read the summaries and access the recordings on-demand.
Session 1 - Opening Keynote: From Automation to Orchestration: Roadmap Update
Giuseppe Damiani, CEO of Stonebranch, kicked off the keynote session of Stonebranch Online on September 28th. After welcoming attendees and offering a few opening remarks, he turned the session over to Stonebranch executives Peter Baljet and Colin Cocksedge to share their perspectives on industry trends and reveal the roadmap for the Stonebranch Universal Automation Center (UAC).
Industry Trends and Transformations
Peter Baljet, CTO of Stonebranch, shared his insights on the continuing evolution of IT automation, covering everything from the growing complexity of the IT landscape, to the democratization of IT to non-technical business users. He pointed out a few key drivers in this evolution, including the:
- Shift to the cloud creates new complexity in running automation across a hybrid IT environment
- Demand for advanced functionality including containers and analytics
- Maturity of DevOps and agile development create new pressures on IT ops teams
- Requirements to centralize and orchestrate automation across siloed on-prem and cloud systems
- Increased LOB needs (like faster access to IT actions) and decreasing IT resources and budgets
- Concern about an aging workforce that currently holds the tribal knowledge around managing legacy environments
Using a service orchestration and automation platform (SOAP) — not to be confused with the SOAP network protocol — helps tackle these trends by going beyond traditional workload automation. SOAPs like the Stonebranch UAC centralize management of off-the-shelf applications (SaaS or on-prem), cloud service providers, data pipelines, DevOps toolchains, robotic process automation, and containerized microservices.
He then turned it over to Colin to discuss the how the UAC will be ready to meet these market shifts.
Stonebranch UAC Roadmap
Colin Cocksedge, Director of Product Management at Stonebranch, presented some exciting platform updates that you can expect to see in the UAC in the short- to medium-term. In the short-term, he highlighted three key development efforts focused on:
- User experience — making the automation experience as good as it can get, both for the infrastructure and operations teams, as well as the citizen automation community
- Data analytics — more (and better) data discovery and visualization capabilities embedded within the UAC to to help you improve observabity and make data-based decisions from UAC’s single pane of glass
- Development platform — rapid, agile development of integrations and extensions to support modern methodologies (jobs-as-code) and offer future-proof scalability
The mid-term roadmap reinforces these areas, and also boosts cybersecurity and AI capabilities.
Colin also showcased the new Integration Hub, which makes it easy to find and download extensions that enable cross-functional, cross-platform orchestration and automations.
This on-demand recorded session is available by request. Because it contains more sensitive roadmap information intended for customers and partners, it’s the only session in the series where we don’t allow public access. If you are a customer or partner and would like to gain access, simply contact us.
Session 2 - Industry Expert Perspective: Eckerson Group — Using DataOps Methodologies to Orchestrate Your Data Pipelines
Kevin Petrie, VP of Research at Eckerson Group, introduced DataOps methodologies and explored how it’s driving significant change for leading organizations.
He made it clear that data pipelines are not an end in themselves. Ultimately, they’re plumbing, and you need to orchestrate all the pipes in order to run an effective business.
In a live poll of the session attendees, nearly half said their greatest challenge in data management is its complexity. Kevin identified streaming data pipelines as a way to improve efficiency in data management; this approach also enables improvements in data science, business intelligence, embedded analytics, IoT maintenenance, cloud migrations, log analytics observability, and security analytics. DataOps is the methodology that makes those pipelines more efficient and effective, through continuous integration / continuous delivery (CI/CD), as well as orchestration, testing, and monitoring.
Pipeline-enabling technologies include cloud data platforms, DataOps platforms, DevOps platforms, data integration tools, and observability tools. Data pipeline orchestration platforms — like the Stonebranch UAC — are critical to bring them all together in harmony.
Scott Davis, VP of Global Marketing at Stonebranch, joined Kevin to highlight just how effective the Stonebranch UAC is as a data pipeline orchestration platform.
More than three-quarters of attendees said that it’s difficult to tell just how many data sources there are across their organizations — there are too many to count. Without a clear orchestration tool to run across your entire data pipeline, you wind up in a scenario where you don’t really have a centralized view into the pipeline. That’s a problem if you want to keep things running smoothly.
The Stonebranch UAC supports data pipeline orchestration using DataOps methodologies, along with a whole bunch of other activities involved in integrating and controlling applications across the data toolchain. It gives you flexibility to code your own workflows or use the no-code designer — whichever you’re most comfortable with. And it can connect to and control any application along the data pipeline.
Watch the full catalog of Stonebranch Online 2021 events on-demand — visit the event overview page to learn more.
Start Your Automation Initiative Now
Schedule a Live Demo with a Stonebranch Solution Expert