In Short: One Fixed Speed, the Other Fixed Waste
Two platform investments explain how Azure Databricks answered its historic criticisms. Photon - a vectorised query engine written in C++ behind standard Spark APIs - fixed speed: lakehouse SQL that once trailed dedicated warehouses became genuinely fast, with no code changes. Serverless compute fixed waste: sessions that start in seconds and bill only while working ended the era of clusters idling expensively between uses.
Together they are why the lakehouse-serves-BI claim became credible, and why Databricks cost conversations changed shape.
Photon, Without the Whitepaper
Traditional engines process rows; Photon processes columns in CPU-friendly batches (vectorised execution), squeezing modern hardware properly. It sits behind the same Spark SQL and DataFrame APIs, so adoption is a checkbox, not a migration.
Where it shines: SQL-shaped work - joins, aggregations, scans over Delta tables - which is to say, BI serving and most transformation pipelines. Where it fades: code that lives in custom Python UDFs, which falls back off the fast path; the optimisation there is moving logic into native operations. And it compounds with the boring disciplines: well-compacted, well-sized files remain the prerequisite - an engine cannot vectorise its way past ten thousand tiny files.
Serverless, and the Economics That Matter
Classic Databricks compute is clusters you size, start and (crucially) remember to stop. Serverless inverts the model: the platform holds warm capacity, sessions attach in seconds, scale with demand and detach when idle - billing follows the work, not the calendar.
The practical wins, in the order clients notice them:
- BI concurrency - serverless SQL warehouses absorb bursty dashboard crowds without an always-on cluster sized for the peak
- Interactive work - analysts and data scientists stop budgeting minutes for cluster starts, and stop leaving clusters running out of start-time dread
- Irregular jobs - schedules with quiet weekends no longer pay for them
The counterweight: steady, saturated batch pipelines on well-managed classic clusters can still be cheaper per unit of work. The mature posture is per-workload measurement - the same discipline as capacity management on the Fabric side, with DBUs in place of capacity units.
What This Means for Platform Decisions
Photon and serverless removed the last easy criticisms in the Databricks-versus-Fabric conversation: speed and idle-cost arguments are largely settled on both sides. What remains decisive is everything else - team skills, governance homes, the BI and AI experience each platform wraps around the same open data. Which is, frankly, a healthier basis for the decision than benchmark bickering ever was.
Sources and Further Reading
Frequently asked
A query engine written from scratch in C++ using vectorised execution, compatible with Spark APIs, that dramatically accelerates SQL and DataFrame workloads without code changes. It is the performance foundation of Databricks SQL and available across much of the platform's compute.
No - that is its selling point: it accelerates existing Spark SQL and DataFrame operations transparently. Workloads dominated by such operations see large gains; heavy custom Python UDF code sees less, because execution falls back outside the vectorised path.
Compute the platform runs and scales for you: sessions start in seconds, scale with load, and bill only while working. It removes cluster management and the classic cost failure of clusters idling between uses. Serverless options cover SQL warehouses, notebooks and jobs.
For spiky, interactive workloads - BI queries, ad-hoc analysis, irregular jobs - usually yes, because idle time disappears. For steady, saturated batch workloads a well-managed classic cluster can still win. The honest answer is measured per workload, not declared platform-wide.
They are why lakehouse SQL performance and consumption-style economics are now table stakes on both platforms: Fabric bills capacity units, Databricks bills DBUs with serverless per-use. The comparison decision moved from raw speed to experience, governance and team fit.
