In Short: Parquet Plus a Log Equals a Real Table
Delta Lake is the answer to an old embarrassment: data lakes stored files, not tables, and files fail in ways databases spent decades eliminating - half-written jobs, readers seeing partial data, no history, no schema enforcement. Delta keeps the cheap, open storage (Parquet files in your object store) and adds a transaction log beside them. The log records every change atomically, and that single idea delivers ACID transactions, consistent reads, versioned history and schema control on top of ordinary cloud storage.
It matters to a Microsoft-centric estate for one strategic reason: it is the shared native format of both OneLake in Fabric and Azure Databricks. The platforms interoperate because the bytes agree.
What the Transaction Log Buys You
- Atomic writes - a job that dies mid-write commits nothing; readers never see half a load
- Consistent reads - queries see the table as of a committed version, even while writers work
- Time travel - previous versions remain queryable and restorable until maintenance removes them: the undo button for the Tuesday someone overwrote the orders table
- Schema enforcement and evolution - writes that do not match the schema fail loudly, and deliberate schema changes are recorded rather than accidental
- Merge and change data - upserts (MERGE) and change data feeds make incremental patterns practical, which is what medallion pipelines are built from
None of these are exotic database features. That is the point: the lake finally behaves like a database while remaining open files you own.
Why One Format Underneath Two Platforms Changes Decisions
Because Fabric and Databricks both speak Delta natively, several previously painful choices become soft.
Coexistence is real. A table Databricks engineering maintains can appear in Fabric via shortcut or Unity Catalog mirroring and serve Power BI through Direct Lake - no export, no copy, no sync job. The platform comparison stops being either-or.
Lock-in drops. Your data's format is open-source and readable by a wide ecosystem. Platform decisions become about engines, experience and governance rather than about where the bytes are trapped.
Skills transfer. Optimisation concepts - compaction, file sizing, V-Order and OPTIMIZE, vacuum and retention - are the same discipline on both platforms, because they are properties of the format.
The Operational Habits Delta Expects
Delta is low-maintenance, not no-maintenance. Small files accumulate with every append and need periodic compaction; time-travel history consumes storage until VACUUM trims it, and retention deserves a deliberate setting rather than a default; and schema evolution is a feature to use consciously, not a surprise to discover downstream. Our optimisation guide covers the Fabric-side specifics, and the same habits apply verbatim on Databricks.
The bottom line for decision-makers: Delta Lake is the safest storage bet in the Microsoft data world precisely because it is not a Microsoft-only bet. Build the estate on open tables, and the platform conversation above them stays a choice rather than a hostage negotiation.
Sources and Further Reading
Frequently asked
An open-source table format: ordinary Parquet data files plus a transaction log that records every change. The log is what turns a folder of files into a reliable table - with atomic writes, consistent reads, versioned history and enforced schema.
Databricks created it; Microsoft adopted it as OneLake's native format. Sharing the format means the two platforms can read each other's tables directly - shortcuts and mirroring between them work precisely because the bytes on disk are the same standard.
The transaction log keeps previous versions of a table, so you can query it as it was at an earlier version or timestamp, and restore it after a bad write. Retention is finite and controlled by table maintenance, so it is an operational safety net rather than an archive.
Delta is the enabling format; the lakehouse is the architecture built on it. ACID tables on cheap object storage are what let one storage layer serve SQL, Spark and BI - the lakehouse promise - without copying data into a separate warehouse.
Apache Iceberg is the other major open format, and the ecosystem is converging on interoperability between the two rather than a winner-takes-all outcome. For Microsoft-centric estates the practical answer is unchanged: Fabric and Databricks speak Delta natively, and openness protects you either way.
