In Short: Data Engineering, Consolidated
LakeFlow is Databricks tidying its data engineering story into one stack: Connect for getting data in (managed, incremental connectors from operational databases and SaaS applications), declarative pipelines for transforming it (the continuation of Delta Live Tables), and jobs for orchestrating everything. The pieces existed in some form before; the consolidation matters because it turns pipeline architecture on Azure Databricks from a build-it-yourself exercise into a set of platform defaults.
For teams deciding how much pipeline machinery to own, that is the real story: the platform now has an opinion, and it is a good one.
Connect: The Ingestion Tier Moves In-House
LakeFlow Connect provides managed connectors that continuously and incrementally ingest from sources - operational databases and common SaaS platforms - straight into Delta tables, monitored and governed inside the platform.
The significance is architectural: the third-party ELT tool that many estates bought purely to feed the lakehouse becomes optional for covered sources. The honest caveats: connector coverage is specific and growing, so audit it against your actual sources, and complex source-side transformations still belong downstream, not in the ingestion hop. If you know mirroring in Fabric, the shape is familiar - replication as configuration, on the Databricks side of the fence.
Declarative Pipelines: Say What, Not How
The centrepiece. Instead of orchestrating notebooks that imperatively build tables, you declare the tables: this silver table is that bronze source, cleaned like so; this gold table aggregates that silver one. The platform derives the dependency graph, runs incrementally, retries, and surfaces lineage and quality.
- Expectations attach data quality rules to definitions: rows failing a rule can be dropped, quarantined or fail the pipeline - quality enforcement as code, where it belongs
- Incremental by default: the platform tracks what changed rather than reprocessing history on faith
- Less glue to own: dependency management, checkpointing and monitoring stop being your codebase
The pattern maps one-to-one onto medallion architecture: declare bronze-to-silver-to-gold, and the pipeline is the architecture diagram, executable.
Jobs: The Orchestrator That Was Already There
LakeFlow jobs are the mature Databricks workflow engine under a consolidated name: schedule and chain notebooks, pipelines, SQL and more, with dependencies, retries and alerts - including tasks that publish to Power BI so the reporting layer updates as part of the pipeline rather than by hope.
For most estates, jobs are the boundary where LakeFlow meets everything else: the nightly run that lands data through Connect, transforms it through declarative pipelines, refreshes the BI layer and pings the on-call channel when reality intervenes.
What We Advise
New builds on Databricks should default to the LakeFlow stack: connectors where coverage exists, declarative pipelines for the medallion, jobs on top - and reserve hand-rolled notebook orchestration for the genuinely unusual. Existing DLT estates are already home. And in dual-platform estates, keep one ingestion owner per source: LakeFlow for what Databricks processes, Fabric's tooling for what Fabric owns, and resist ingesting the same source twice because two teams each had a favourite tool.
Sources and Further Reading
Frequently asked
Databricks' unified brand for data engineering: LakeFlow Connect (managed connectors ingesting from databases and SaaS applications), declarative pipelines (the evolution of Delta Live Tables, where you declare transformations and the platform manages execution), and jobs for orchestration across all of it.
A pipeline model where you write the transformations - as SQL or Python defining tables from other tables - and the platform works out dependency order, incremental processing, retries and monitoring. Less orchestration code to own, with data quality expectations built into the definitions.
The technology continued as LakeFlow's declarative pipelines: same core ideas, consolidated branding and evolving features. Existing DLT knowledge transfers directly.
The third-party ingestion tier for supported sources: managed, incremental connectors from operational databases and SaaS apps into Delta tables, run and monitored inside Databricks rather than in a separate ELT tool. Coverage is source-specific, so check the current connector list against your estate.
Same job, different homes: managed connectors plus pipelines plus orchestration. Fabric's version leans low-code and integrated with the wider SaaS platform; LakeFlow leans code-first with engineering ceremony. Estates running both platforms typically ingest with whichever owns that source's downstream processing.
