Orchestration, Observability, and Control Over the Hybrid Data Pipeline

Modern enterprise data pipelines don't live in one place. They span mainframes, SAP systems, Azure, AWS, on-premise databases, and SaaS platforms — with different schedulers embedded in each layer. Stitching it together without losing visibility is one of the hardest operational problems in data engineering.

This TDWI-sponsored webinar tackles that problem from two angles: research-backed industry analysis and a live product demo.

James Kobielus, Senior Research Director for Data Management at TDWI, opens with findings from TDWI survey research on the top challenges enterprises face managing data in cloud and hybrid environments. Key findings: managing data integration across hybrid environments is the top challenge, close to three-quarters of data professionals agree that AI/ML tools are critical for pipeline management, and an equal number agree that automation is essential — yet satisfaction with current self-service tooling is split almost evenly.

Peter Baljet, CTO at Stonebranch, follows with a framework for thinking about orchestration as a solution to data pipeline tool sprawl. He covers how orchestration tools differ from Airflow, cloud-native schedulers, and legacy on-premise tools, why centralized observability through OpenTelemetry matters, and how treating automation as code through a dev-test-prod lifecycle is increasingly essential for mission-critical pipelines.

Nils Buer then demos a live end-to-end data pipeline in Universal Automation Center. The workflow ingests sales data from Azure Data Lake Gen2, AWS S3, a cloud database, SharePoint, and an IBM mainframe — all streamed directly to a central S3 data lake without intermediate local storage. From there, three parallel streams run: SAP commission calculations, an Azure Synapse machine learning pipeline approved via Slack, and an AWS Glue job loading data into Amazon Redshift for SQL reporting. The workflow closes with a Power BI dashboard refresh confirming the new products appear with live data.

Incident handling is also demonstrated: a failed SAP job automatically creates a PagerDuty ticket with the error log attached. The operations team corrects the credentials and reruns directly from PagerDuty. A Grafana observability dashboard is shown alongside, monitoring task execution history, late-finish SLA violations, file transfer metrics, SQL query revenue outputs, and server health — all without requiring anyone to log into the controller.

The panel discussion covers the future of enterprise data pipelines, how generative AI fits into automation code generation via JSON/YAML representations, and practical advice for small teams starting from on-premise manual processes.

Key Highlights

• TDWI research: managing hybrid data pipeline integrations is the top cloud data challenge for enterprises
• Framework: why Airflow, cloud-native schedulers, and legacy tools all fall short as an orchestration layer
• Live demo: event-driven data ingestion from Azure, AWS S3, SharePoint, and IBM mainframe into a central data lake
• Parallel pipeline streams: SAP commissions, Azure Synapse ML, AWS Glue to Redshift, Power BI refresh
• Slack approval gate for cost-intensive Synapse pipeline before execution
• Automated PagerDuty incident creation with SAP error log, with rerun triggered from PagerDuty
• Grafana observability dashboard: task metrics, SQL data quality checks, transfer rates, server health via OpenTelemetry
• Panel discussion: GenAI for automation code generation, crawl-walk-run adoption advice, UAC as SaaS for hybrid environments

About the Presenters

James Kobielus, Senior Director of Research, Data Management, TDWI
James Kobielus is a veteran industry analyst, author, and speaker with three decades of experience across data management, analytics, and AI. He has held analyst positions at Futurum Research, Wikibon, Forrester Research, Current Analysis, and The Burton Group, and served as Senior Program Director for Big Data Analytics at IBM. At TDWI, he focuses on data management including data governance, data integration, master data management, and DataOps pipelines.

Peter Baljet, CTO, Stonebranch
Peter Baljet brings more than 25 years of experience in technology, product strategy, and engineering to his role as CTO at Stonebranch. He is a recognized thought leader in cloud and big data analytics, and previously held senior leadership roles at SAP, Deloitte Consulting, and Visiprise.

Nils Buer, Director of Product Strategy (EMEA), Stonebranch
Nils Buer brings more than 20 years of enterprise IT experience to his work at Stonebranch, with deep expertise in IT automation and orchestration across SAP, cloud platforms, and hybrid IT environments. Before joining Stonebranch, he held senior leadership roles at Ericsson, SAP, BMC, and Capgemini.