In Short: Automated Routes With Someone on Call
A data pipeline is the automated route between where data is born - the ERP, the CRM, the API, the nightly file - and where it earns its keep: the lakehouse, the semantic model, the dashboard, the ML feature table. Along the route: extraction from the source, movement, transformation into analytical shape, and - the part that separates professionals from scripts - orchestration, monitoring and an owner who hears about failures before the business does.
Every number anyone trusts arrived through one. Which is why pipeline quality is quietly the ceiling on everything downstream.
The Anatomy
- Extraction: reading from sources - databases, APIs, files, streams - incrementally where possible, because full reloads are the brute force that stops scaling
- Movement: landing data in the analytical platform, raw first in the ELT style that modern estates default to
- Transformation: cleaning, conforming and shaping through medallion layers until the data answers questions
- Orchestration: the schedule and dependency logic - run this after that, retry on failure, stop the chain when a step fails rather than loading garbage downstream
- Monitoring and alerting: knowing it ran, knowing it was complete, telling a human when either is false
The last two are where estates actually differ. Anyone can move data on a good day; pipelines are judged on the bad ones.
What Good Pipelines Share
They assume sources lie. Columns change, files arrive malformed, APIs throttle. Good pipelines validate what arrives, fail loudly and specifically, and never silently load partial data - the failure mode behind half the refresh horror stories we investigate.
They are incremental. Processing what changed, not everything, every time: cheaper, faster, and the only pattern that survives growth.
They are idempotent. Re-running yesterday's failed load must not double yesterday's numbers. This one property, designed in early, prevents a whole genre of incident.
They are observable and owned. Run history, row counts, freshness metrics visible somewhere; alerts routed to a named owner; and the estate documented well enough that the owner's holiday is not an outage risk.
Pipelines in the Modern Microsoft Stack
The 2026 twist: the best pipeline is often no pipeline. Mirroring replicates operational databases into OneLake with nothing to author; shortcuts reference data without moving it. What remains for authored pipelines - Fabric Data Factory pipelines and Dataflows Gen2, notebooks for code-first transformation, LakeFlow on the Databricks side - is the genuinely bespoke: files, APIs, complex transformation, orchestration across all of it.
So the modern estate runs fewer, better pipelines: replication and virtualisation for the routine, engineered routes for the rest, everything monitored, everything owned. If your current estate is instead a hundred scheduled mysteries, that consolidation - not more tooling - is the modernisation worth funding.
Sources and Further Reading
Frequently asked
An automated route that moves data from where it is created (operational systems, files, APIs) to where it is used (analytics platforms, reports, models), transforming it along the way. Extraction, movement, transformation, and the scheduling and monitoring that keep it running - that whole assembly is the pipeline.
ETL/ELT describes the pattern inside a pipeline (when transformation happens); the pipeline is the whole operational artefact: the steps plus orchestration, retries, alerting and ownership. You run pipelines; ETL and ELT are how they are shaped.
Upstream change: a source system alters a column, credentials expire, an API throttles, a file arrives malformed. Good pipelines assume this - validating inputs, failing loudly, alerting owners - because the question is never whether sources change, only when.
Fewer, which is the point: mirroring replicates databases without authored pipelines, shortcuts reference data without moving it. Pipelines remain for what those cannot do - transformation, complex sources, files and APIs - so the modern estate runs fewer, better pipelines rather than none.
A named person or team per pipeline, accountable for its failures and its data's quality. Orphaned pipelines - running, unowned, half-understood - are how estates decay; the ownership list is as important as the code.
