Microsoft Fabric’s new Native Execution Engine promises to make big data analytics on Spark dramatically faster and cheaper for enterprises, without forcing developers to rewrite a single line of code.
Why Microsoft Fabric built a Native Execution Engine
Modern analytics teams are pushing Spark harder than ever, with bigger datasets, tighter SLAs, and expectations for near real‑time insights. Pipelines that were designed years ago are now running more frequently, on higher data volumes, and often cost more just to maintain the same business outcomes.
On traditional Spark, developers often respond by endlessly tuning the same transformations or scaling up clusters simply to hit existing SLAs. Spark’s JVM‑based execution model also brings limits: garbage collection pauses, row‑oriented execution that fights against columnar formats like Parquet and Delta Lake, and minimal use of SIMD instructions on modern CPUs. As workloads scale, even highly optimized Spark runtimes begin to hit practical performance ceilings, driving the need for a new execution approach that squeezes more from existing hardware without breaking the Spark developer experience.
Microsoft’s answer inside Fabric is the Native Execution Engine—a C++‑powered execution layer that offloads the hottest parts of Spark queries into a vectorized, columnar engine while leaving planning, scheduling, and fault tolerance in the familiar Spark stack.
What the Native Execution Engine is and how it works
Fabric’s Native Execution Engine introduces a new execution path for Spark that pushes supported operators into a high‑performance C++ engine instead of the traditional JVM row‑based stack. Spark still handles the logical plan, physical plan, distribution, and fault tolerance, but the heavy lifting for supported operations happens natively in C++.
Two open‑source technologies sit at the core of this design:
-
Velox – Meta’s vectorized C++ execution engine that provides optimized kernels over columnar data, enabling batch‑oriented, SIMD‑friendly processing.
-
Apache Gluten – An Apache incubating project that bridges Spark and Velox by translating Spark’s physical plan into a native plan Velox can execute.
In practice, Spark builds its logical and physical plans as usual, then Gluten inspects the physical plan and replaces operators that it knows how to execute with native equivalents while leaving unsupported operators to run on the regular Spark engine. Velox then executes those native operators over columnar batches using vectorized kernels.
Because the Native Execution Engine hooks in after Spark’s logical and physical optimization phases, it preserves all the existing Fabric Spark optimizations—adaptive query execution, cost‑based rewrites, column pruning, predicate pushdown, and more. The data engineer still writes standard Spark code and notebooks; turning on native execution is essentially a configuration change rather than a rewrite.
Columnar execution, SIMD, and why it’s faster
Traditional Spark execution is row‑oriented: values for a single row are stored together in memory, which is flexible but inefficient for analytics, where the same operation gets applied across millions of values in a column. With row‑based layouts, the CPU has to jump around memory to access different columns, which leads to cache misses and limited parallelism.
The Native Execution Engine adopts a columnar layout, storing values for each column contiguously, which improves memory locality and makes it natural to process data in vectors rather than row by row. Paired with SIMD (Single Instruction, Multiple Data) instructions, the CPU can apply a single operation—like a comparison, arithmetic function, or hash—across many values at once, often 8, 16, or even 32 elements depending on the hardware.
Velox exploits this design by executing queries over columnar batches through highly optimized vectorized operators. This increases cache efficiency, reduces branching, and boosts throughput for projections, filters, aggregations, and expression evaluations. Running these operators in C++ also avoids JVM JIT warm‑up, reduces serialization overhead, and gives the engine fine‑grained control over memory allocation and layout—further squeezing performance out of the same underlying infrastructure.
Where the Fabric Native Execution Engine shines
Microsoft is clear that the Native Execution Engine is not a silver bullet for every Spark workload, but when used in the right scenarios the gains are dramatic. The engine focuses on CPU‑heavy, analytics‑style queries and currently works best in a few key patterns:
-
Parquet and Delta Lake data – The engine can process Parquet and Delta formats natively, avoiding unnecessary conversions and taking full advantage of columnar storage.
-
Complex transformations and aggregations – Workloads with heavy grouping, filtering, projections, and expression evaluation benefit most from vectorized operators.
-
Queries that avoid fallbacks – Staying within the set of supported operators and expressions avoids bouncing between native and JVM execution, which helps maintain throughput.
-
Compute‑intensive workloads – CPU‑bound analytics benefit more than simple scans or jobs dominated by I/O, where disk or network remains the bottleneck.
On benchmark workloads, Microsoft is reporting large gains: running the TPC‑DS benchmark at scale factor 1000 on Delta format shows up to 6x faster performance compared to open‑source Spark. On a fixed‑size Fabric cluster, this kind of speedup translates to roughly 83% compute‑cost savings for the same workload.
A more practical example comes from the NYC Yellow Taxi dataset, where a SQL query computing per‑vendor aggregates was run twice—once with the Native Execution Engine disabled and once with it enabled through the spark.native.enabled configuration. With native execution on, the query completed about 2.4x faster than the baseline run.
Handling fallbacks and getting visibility in Spark notebooks
The Native Execution Engine does not yet support every Spark operation, so some queries will partially or fully fall back to JVM‑based execution. When this happens, data needs to move between columnar and row formats at the native/JVM boundary, which can degrade performance and make it harder to understand why a job isn’t as fast as expected.
To address that, Microsoft has enhanced the Apache Spark advisor in Fabric to surface real‑time, in‑context alerts whenever an operator falls back from native to JVM execution. These alerts appear during notebook cell execution and inside the Spark Advisor view, helping engineers quickly identify which parts of a workload are not being accelerated natively.
The blog highlights a concrete example: a query that uses .show() triggers a fallback because .show() internally calls collectLimit and toprettystring, which are not yet supported by the Native Execution Engine. Swapping .show() for .collect() keeps the query on the native path and eliminates the fallback. This kind of small refactor can make a significant difference when repeated across frequently executed jobs.
With this visibility, teams can look for recurring fallbacks, refactor expressions to supported equivalents, keep data columnar for longer, and then re‑run workloads to confirm that more of the pipeline is being executed natively.
Enabling the Native Execution Engine in Fabric
One of the biggest selling points of the Native Execution Engine is that it delivers performance gains with no code changes, and at no additional compute cost on Fabric. Teams don’t have to rewrite existing notebooks or pipelines—they simply turn on the native path and let the engine accelerate supported operators.
You can enable the Native Execution Engine globally at the environment level using Fabric’s acceleration settings. Within Spark application code, you can also toggle it on or off using the documented configuration options for native execution, which is useful for A/B testing performance and validating impact on specific jobs.
Microsoft recommends running the Apache Spark run series to compare behavior and performance before and after enabling native execution. Complementary tools like the Apache Spark advisor, DataFrame explain, and Spark History Server can then be used to inspect plans, understand where native execution is kicking in, monitor fallbacks, and validate that workloads are behaving as expected.
According to Microsoft, now is a good time for Fabric customers to start enabling the Native Execution Engine, observe the resulting performance improvements, and iteratively tune Spark pipelines to take advantage of native, vectorized execution.
Recent Posts You Might Like
- Microsoft MFA Outage Today: Azure, Outlook, and Microsoft 365 Hit With 504 Gateway Errors
- Microsoft Sentinel Just Changed How Security Teams Automate Threats in 2026 — AI SOAR Playbooks Are Here
- New Microsoft 365 OAuth Hijacking Campaign Dangerously Bypasses MFA: What Admins Must Do Now
- Microsoft Sovereign Cloud Can Now Run Big AI Models Fully Disconnected — Here’s Why It Matters
About The Author
Discover more from Microsoft News Now
Subscribe to get the latest posts sent to your email.
