Fabric Runtime 2.0: Spark 4.1, Delta 4.2 and What to Test Before Upgrading

In Short: What Fabric Runtime 2.0 Changes

Fabric Runtime 2.0 is the new Spark runtime for Microsoft Fabric's data engineering and data science workloads. It reached general availability in August 2026 and Microsoft Learn now lists it as the default runtime for new workspaces and environment items. It moves Fabric to Apache Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21 and Scala 2.13 on Azure Linux 3.0.

Several of those upgrades cross major-version boundaries, so this is not a switch to flip on a Friday afternoon. Existing workspaces stay on their current runtime until you change them, and Runtime 1.3 is already on borrowed time. This post covers what changes, what is most likely to break, how to choose a runtime per workspace and environment, and the upgrade and testing plan we use. If Spark itself is new to you, start with our introduction to Apache Spark.

What Is in Runtime 2.0

Microsoft Learn lists these components, with Runtime 1.3 for comparison:

• Apache Spark - 4.1 (was 3.5.5)
• Delta Lake - 4.2 (was 3.2)
• Python - 3.13 (was 3.11)
• Java - 21 (was 11)
• Scala - 2.13 (was 2.12)
• R - 4.5.2 (was 4.4.1)
• Operating system - Azure Linux 3.0 (was Azure Linux 2.0)

Spark 4.1 brings new SQL features such as the VARIANT data type, SQL user-defined functions, session variables, pipe syntax and string collation. On the PySpark side there is a native plotting API, a Python Data Source API and Python user-defined table functions. On the Delta side, Microsoft's support matrix shows Runtime 2.0 adding type widening, which Runtime 1.3 does not support.

Runtime 2.0 also supports the native execution engine, Fabric's vectorised C++ engine built on Apache Gluten and Velox. More on that below.

Why This Needs a Plan Now

Microsoft's runtime lifecycle page lists Runtime 1.3 as "end of support announced", with an end of support date of 30 September 2026. It entered Long Term Support on 1 October 2026 for six months, which extends support through March 2027. Runtime 2.0 is supported until 31 August 2028.

The lifecycle page is blunt about what end of support means: the runtime is removed from workspace settings and environment items, and "existing Spark jobs running on the existing environments can't run." Microsoft also notes that lifecycle timelines can change.

So the real deadline for most estates is March 2027. That is five months away, and a careful migration of a busy lakehouse takes weeks, not days.

What Can Break

Microsoft's migration guide is unusually candid. These are the areas we check first.

Libraries and Python 3.13

Runtime 2.0 ships a leaner default package set. The guide lists 348 Conda packages and 24 pip packages removed from the default Python environment, including familiar names such as dask, numba, pytorch, transformers, shap and xlsxwriter. If a notebook imported one of these without you ever installing it, it will fail on Runtime 2.0 until you add it to an environment.

Every Python package also needs a Python 3.13-compatible release, and packages with native extensions need a Linux wheel that works on Azure Linux 3.0. Spark 4.1 raises the minimum versions of pandas, NumPy and PyArrow and removes deprecated pandas API on Spark calls such as `DataFrame.append` and `iteritems`.

Java, Scala and custom JARs

Scala 2.13 is not binary compatible with Scala 2.12. A Runtime 1.3 JAR can fail with class-loading errors on Runtime 2.0 even if the source has not changed. Custom JARs need recompiling against Java 21, Scala 2.13 and Spark 4.1, and Maven dependencies need their `_2.13` versions.

Spark settings and SQL behaviour

Fabric keeps ANSI SQL mode off in Runtime 2.0, even though upstream Spark 4.1 turns it on. That avoids one large source of breakage. Other defaults do change. A few that affect file sizes and performance:

• `spark.sql.files.maxPartitionBytes` rises from 128 MiB to 512 MiB
• `spark.hadoop.parquet.block.size` falls from 1 GiB to 128 MiB
• adaptive target file sizing and optimised writes for partitioned tables are switched on
• `OPTIMIZE` can now skip work when compaction criteria are not met

None of these is wrong, but each can change file layout, parallelism and run times. Compare schemas, row counts, nulls, casts and date and timestamp results between runtimes rather than assuming they match. Our guide to Spark and Delta optimisation in Fabric explains why file sizes matter so much downstream.

Delta Lake tables and protocol versions

This is the one we worry about most, because a mistake here is hard to reverse. Runtime 2.0 can read and write tables created on Runtime 1.3. But new tables created on Runtime 2.0 default to reader version 3 and writer version 7 with deletion vectors enabled, where Runtime 1.3 defaults to 1 and 2.

That matters because not every Fabric experience supports every feature. Microsoft's interoperability matrix shows Python notebooks (which use delta-rs rather than Spark) and pipelines without deletion vector support, while the SQL analytics endpoint and Direct Lake semantic models do support it. Delta Lake 4.x features such as collations and the variant type are experimental and limited to Spark notebooks and jobs.

Microsoft's guidance is clear: during the migration, don't upgrade table protocols or enable Delta Lake 4.2 features, check the protocol of newly created tables, and list every engine that reads or writes each table. A protocol upgrade "isn't generally reversible".

Streaming

The deprecated `EventHubConnector` is not available in Runtime 2.0; Microsoft points you to the Spark Kafka connector against the Event Hubs Kafka endpoint. Test that each Structured Streaming query can restart from a copy of its existing checkpoint before you move it.

The Native Execution Engine on Runtime 2.0

The native execution engine is the most tangible reason to move. Microsoft's benchmark (TPC-DS at scale factor 1000 on Delta) shows up to six times faster performance than open-source Spark, which it equates to roughly 83% compute-cost savings on a fixed-size cluster. Your workloads will differ, so treat that as a ceiling, not a forecast.

Three practical points:

• It is not on by default. You enable it on the Acceleration tab of an environment, or with `spark.native.enabled` for a single notebook or job.
• Runtime 2.0 fixes several correctness gaps that applied on Runtime 1.3, including rounding, decimal-to-float casts and `collect_list` ordering, and supports ANSI mode natively.
• Unsupported work falls back to JVM Spark. Structured Streaming, JSON and XML are not accelerated. Fabric Spark Advisor flags fallbacks in notebook output.

The migration guide also records a production rollout on 16 October 2026 that enabled cost-based fallback, so recheck query plans for workloads you tested before that date.

Choosing a Runtime per Workspace and Environment

There are two levels, and the difference matters:

• Workspace default - set in Workspace settings > Data Engineering/Science > Spark settings > Environment. It applies to notebooks, Spark job definitions and lakehouse operations from the next Spark session. Running sessions carry on with their current runtime.
• Environment item - choose the runtime in the environment, save and publish, then attach the environment to specific notebooks or Spark job definitions. An environment's runtime overrides the workspace default.

That gives a clean way to migrate in pieces. Leave the workspace on Runtime 1.3, create a Runtime 2.0 environment, and move items to it one batch at a time. Only switch the workspace default once every item has moved. When you change an environment's runtime, Fabric migrates the library list, but publishing fails if dependencies conflict, and JARs often need replacing.

Fabric also offers runtime release channels (in preview). Setting `spark.computeConf.runtime.releaseChannel=earlyAccess` in a test environment lets you run against the next runtime update before it becomes the default. Early access needs a custom pool rather than the starter pool, so sessions start more slowly.

A Safe Upgrade and Testing Plan

This follows Microsoft's staged approach, with the checks we add from experience.

• 1. Inventory. List notebooks, Spark job definitions, schedules, environments, Spark settings, libraries with resolved versions, custom JARs and wheels, Delta tables and every non-Spark consumer of them, and streaming checkpoints. Record baseline row counts, schemas and run times.
• 2. Build a Runtime 2.0 test environment. Create a separate environment item in a development workspace, add the libraries you need with pinned versions, and publish it. Our branch workspaces and deployment guide covers setting this up without touching production.
• 3. Fix code and dependencies. Add removed packages, rebuild JARs, update removed PySpark calls, replace the Event Hubs connector and remove unsupported Spark settings. Leave Delta table protocols alone.
• 4. Run both runtimes side by side. Feed the same inputs to Runtime 1.3 and Runtime 2.0 and compare outputs, schemas, Delta history and protocol, run times and capacity use. Check results in the SQL analytics endpoint and Power BI too, not just in Spark. Include failure tests: retries, session restarts and checkpoint recovery.
• 5. Roll out in batches. Attach the validated environment to a small set of production items, watch a full cycle of schedules, then continue. Our guide to workspace monitoring and observability covers how to spot regressions in duration and capacity use quickly.
• 6. Keep a way back. Keep the Runtime 1.3 environment, artefacts and checkpoints until the observation period ends. Rolling back means reattaching the old environment and starting a new session - until March 2027, when that option goes.

Only after everything has run cleanly would we consider enabling the native execution engine broadly, and later still any Delta 4.2 table features.

Where Solv Systems Comes In

We run Runtime 2.0 migrations as a short, fixed-scope piece of work: an inventory of your Spark estate, a dependency and Delta compatibility review, side-by-side test runs, and a batched cut-over with rollback. Our Microsoft Fabric consultants also benchmark the native execution engine on your own workloads, so you know what it saves before you rely on it.

Sources and Further Reading

• Runtime 2.0 in Fabric (Microsoft Learn)
• Migrate from Fabric Runtime 1.3 to Runtime 2.0 (Microsoft Learn)
• Apache Spark runtime lifecycle in Fabric (Microsoft Learn)
• Delta Lake table format interoperability (Microsoft Learn)
• Native execution engine for Fabric Data Engineering (Microsoft Learn)
• Fabric runtime release channels (Microsoft Learn)