Apache Auron™Apache Auron™
Introduction
  • Getting Started
  • Configuration
  • Spark Memory Management
  • Benchmarks
  • Become a Committer
  • Become a PMC Member
  • Release Guide
  • v9.0.0
  • v8.0.0-incubating
  • v7.0.0-incubating
  • v6.0.0-incubating
  • Pre-ASF Releases

    • v5.0.0
    • v4.0.1
    • v4.0.0
  • All Archived Releases
  • License
  • Security
  • Privacy
  • Thanks
  • Sponsorship
  • Events
  • Foundation
GitHub
Introduction
  • Getting Started
  • Configuration
  • Spark Memory Management
  • Benchmarks
  • Become a Committer
  • Become a PMC Member
  • Release Guide
  • v9.0.0
  • v8.0.0-incubating
  • v7.0.0-incubating
  • v6.0.0-incubating
  • Pre-ASF Releases

    • v5.0.0
    • v4.0.1
    • v4.0.0
  • All Archived Releases
  • License
  • Security
  • Privacy
  • Thanks
  • Sponsorship
  • Events
  • Foundation
GitHub
  • Introduction
    • Overview
    • Introduction
    • Architecture
    • Join the Community
  • Documents

    • Getting-Started
    • Configurations
    • Spark Memory Management
    • Benchmarks
  • Community

    • Become a Committer
    • Become a PMC Member
    • Release Guide
  • Archives

    • v9.0.0
    • v8.0.0-incubating
    • v7.0.0-incubating
    • v6.0.0-incubating
    • Pre-ASF Releases

      • v5.0.0
      • v4.0.1
      • v4.0.0
    • All Archived Releases

Apache Auron 9.0.0

Apache Auron 9.0.0 is the first release after Auron graduated as an Apache Software Foundation top-level project. This release expands native Spark and Flink execution, strengthens lakehouse scan support, introduces memory-bounded execution improvements, and delivers broad correctness, observability, build, and runtime-stability enhancements.

New Features

  • Spark 4.2 support: Added build and runtime compatibility for Apache Spark 4.2.
  • Memory-bounded native joins: Added hybrid hash-join fallback for large build sides and expanded residual-condition support across SortMergeJoin, BroadcastHashJoin, ShuffledHashJoin, and BroadcastNestedLoopJoin.
  • Expanded Iceberg native scans: Added runtime filters, _partition metadata, identity partition values, changelog pruning, decimal residual pruning, and improved schema-evolution support.
  • Native expressions and functions: Added native support for additional array and datetime functions, including array_contains, array_distinct, array_union, arrays_overlap, datediff, date_add, date_sub, add_months, next_day, trunc, and coalesce.
  • Flink operator fusion: Added source–Calc fusion into a single native plan and native support for UNIX_TIMESTAMP.
  • Native conversion diagnostics: Added operator conversion status, fallback reasons, AQE snapshots, effective configuration, and live SQL metrics to the Auron UI.

Improvements

  • Memory management: Introduced coordinated native spilling with configurable watermarks and bounded spill concurrency, while honoring Spark executor memory-overhead settings.
  • Execution performance: Optimized primitive sorting and Top-N, array/map explode, batch serialization, IPC compression, broadcast serialization, and aggregate processing.
  • Scan performance: Added bounded parallel reading for small files and improved Iceberg partition-size estimation, pruning, and scan metrics.
  • Runtime stability: Made native initialization retryable, improved spill error handling, and cleaned up native Arrow shuffle files when shuffles are unregistered.
  • Build and packaging: Switched the native engine to stable Rust, added Docker Compose support, fixed ARM Docker builds, and improved Maven option handling.
  • Dependency maintenance: Upgraded DataFusion from 49.0.0 to 55.2.0 and Arrow/Parquet from 55.2.0 to 59.2.0, with reproducible vendor-and-patch management for customized dependencies.

Bug Fixes

  • Fixed Spark semantic differences involving float ordering, float-to-decimal casts, negative square roots, datetime functions, nullable structs, and string casts.
  • Fixed aggregate correctness issues involving filtered aggregates, sliced batches, accumulator rows, and placeholder evaluation.
  • Fixed join correctness issues involving residual conditions, complex NULL keys, row counts, build-side fallback, and forced ShuffledHashJoin.
  • Fixed SortExec schema mapping, Top-K, WindowGroupLimit, union partition mapping, NativeCollectLimit, and zero-column shuffle behavior.
  • Fixed native JSON, Protocol Buffers, Arrow FFI, LZ4_HADOOP Parquet, ORC predicate pushdown, and Parquet statistics handling.
  • Fixed Iceberg changelog projection, nested schema evolution, metadata columns, partition metrics, scan locality, and unsupported-data-type fallback behavior.
  • Fixed Spark UDF synchronization, shuffle cleanup, spill-file handling, and native environment initialization failures.

Notable Changes

  • Apache top-level project release: Removed the incubating designation and updated project and release infrastructure following Auron’s graduation.
  • Safer large joins: Hybrid hash-join fallback is enabled by default to reduce memory pressure and avoid expensive full sort-merge fallback for large build sides.
  • Stable Rust toolchain: Native-engine builds no longer require a pinned nightly Rust toolchain.
  • Datetime conversion default: Native datetime-function conversion is disabled by default. Set spark.auron.enableNativeDateTimeFunctions=true to restore the previous behavior.
  • String-cast behavior: spark.auron.cast.trimString now defaults to false. Set it to true if applications rely on automatic whitespace trimming before casts.
  • Spark 4.2 requirements: Spark 4.2 builds require JDK 17 or later and Scala 2.13. Iceberg integration is not currently supported with Spark 4.2.

What's Changed

  • Bump bytesize from 2.3.1 to 2.4.2 by @dependabot[bot] in #2382
  • [AURON #2373] support Docker Compose in build script by @zhaohehuhu in #2374
  • [AURON #2375] Fix Iceberg changelog scan field-id projection by @lyne7-sc in #2376
  • Bump bytes from 1.11.1 to 1.12.1 by @dependabot[bot] in #2383
  • Bump prost-reflect from 0.16.4 to 0.16.5 by @dependabot[bot] in #2384
  • Bump regex from 1.12.3 to 1.13.0 by @dependabot[bot] in #2385
  • [AURON #2265] Add AI-assisted development guidance by @ShreyeshArangath in #2387
  • [AURON #2378] Support runtime filters in native Iceberg scan by @lyne7-sc in #2379
  • Bump regex from 1.13.0 to 1.13.1 by @dependabot[bot] in #2389
  • Bump lz4_flex from 0.13.1 to 0.14.0 by @dependabot[bot] in #2388
  • [AURON #2396] [RELEASE] Bump the development version to 9.0.0-SNAPSHOT by @slfan1989 in #2397
  • [AURON #1865] Merge native operators: fuse Flink source and Calc into one native plan by @weiqingy in #2327
  • [AURON #2352] Lock spills before new spiller by @wForget in #2350
  • Bump serde from 1.0.228 to 1.0.229 by @dependabot[bot] in #2403
  • [AURON #2398] Fix Maven -D argument handling in auron-build.sh by @slfan1989 in #2399
  • Bump futures-util from 0.3.32 to 0.3.33 by @dependabot[bot] in #2401
  • Bump async-trait from 0.1.89 to 0.1.91 by @dependabot[bot] in #2400
  • Bump tokio from 1.52.3 to 1.53.0 by @dependabot[bot] in #2402
  • [AURON #2372] Fix Docker build on ARM hosts by @zhaohehuhu in #2371
  • Bump actions/labeler from 6 to 7 by @dependabot[bot] in #2410
  • Bump futures from 0.3.32 to 0.3.33 by @dependabot[bot] in #2411
  • Bump tokio from 1.53.0 to 1.53.1 by @dependabot[bot] in #2412
  • [AURON #2413] fix(ci): adapt labeler.yml to actions/labeler v7 by @weiqingy in #2414
  • [AURON #2320] Fix PB deserialization bug & improve PB parsing performance by @Tartarus0zm in #2339
  • [AURON #2417] log rss write size metrics by @cxzl25 in #2418
  • [AURON #1891] Implement randn() function by @robreeves in #1938
  • Bump base64 from 0.22.1 to 0.23.0 by @dependabot[bot] in #2420
  • [AURON #2427] Report fallback reason for unsupported Iceberg scan data types by @weimingdiit in #2428
  • [AURON #2425] Support Iceberg STARTS_WITH pruning in native scan by @weimingdiit in #2426
  • [AURON #2407] [BUILD] Fix the invalid Spark 4.1 Docker build example. by @slfan1989 in #2408
  • [AURON #2329] Test and document multi-consumer source-Calc fan-out safety by @weiqingy in #2415
  • [AURON #2432] Vendor third-party dependencies via setup-vendors.sh with patch series by @diliulou in #2433
  • Bump actions/stale from 10 to 11 by @dependabot[bot] in #2436
  • [AURON #2440] Consolidate Maven properties and format code by @cxzl25 in #2441
  • [AURON #2439] Add git to yum install by @cxzl25 in #2442
  • [AURON #2419] RowNullChecker may mis-detect null join keys for complex Arrow row types by @weimingdiit in #2424
  • [AURON #2429] Add mixed-format fallback coverage for Iceberg changelog native scan by @weimingdiit in #2430
  • [AURON #2446] Flush buffered data based on row count by @zuston in #2447
  • [AURON #2422] Make RSS spill memory triggers configurable by @cxzl25 in #2423
  • [AURON #1863] Support native Flink UNIX_TIMESTAMP: native scalar function by @weiqingy in #2409
  • [AURON #2434] Support full-data-file Iceberg changelog deletes by @weimingdiit in #2435
  • [AURON #2449] Add fallback coverage for unsupported Paimon scan data type by @weimingdiit in #2450
  • [AURON #2453] Fix filtered aggregation for groups with no matching rows by @lyne7-sc in #2454
  • [AURON #1863] Support native Flink UNIX_TIMESTAMP: converter integration by @weiqingy in #2448
  • [AURON #1863] Support the zero-argument Flink UNIX_TIMESTAMP natively by @weiqingy in #2465
  • [AURON #2463] Fix native Top-K with only sort keys by @lyne7-sc in #2464
  • [AURON #2437] Support inner residual join conditions in native broadcast hash join by @weimingdiit in #2438
  • [AURON #2451] Add Paimon COW multi-commit correctness coverage by @weimingdiit in #2452
  • [AURON #2457] Fix per-partition job submission in NativeCollectLimit by @lyne7-sc in #2458
  • [AURON #2431] Prune Iceberg changelog tasks by metadata predicates by @goutamadwant in #2456
  • [AURON #1902] Reuse extension function implementations by @goutamadwant in #2467
  • [AURON #2455] Run UNIX_TIMESTAMP natively for fixed-offset session time zones by @weiqingy in #2466
  • [AURON #2468] Support binary sort keys in native range shuffle by @weimingdiit in #2469
  • [AURON #2472] Fix nullable struct field null propagation by @lyne7-sc in #2473
  • [AURON #2392] Support Spark 4.2 by @XorSum in #2404
  • [AURON #2405] Add primitive fast path for single-column integer sort by @yew1eb in #2406
  • [AURON #2369] fix incorrect ORC predicate pushdown with OR by @Flyangz in #2370
  • [AURON #2482] handle optional sort indices for zero-column batches by @lyne7-sc in #2483
  • [AURON #2474] Fix make_date null and ANSI semantics by @Sigma-Ma in #2479
  • [AURON #2444] Add build branch and build reversion info by @zhaohehuhu in #2445
  • [AURON #2308] Fix native shuffle exchange NPE in shuffle write tasks by @Sigma-Ma in #2487
  • [AURON #2480] Optimize array and map explode using offset ranges by @lyne7-sc in #2481
  • [AURON #2485] Support last_day function natively by @Sigma-Ma in #2486
  • [AURON #2493] Docs: update GitHub repo About description by @diliulou in #2493
  • [AURON #2495] Docs: update README description by @diliulou in #2495
  • [AURON #2488] Support dayofyear function natively by @Sigma-Ma in #2489
  • [AURON #2503] Rename blaze to auron by @Deegue in #2504
  • [AURON #2475] Fix float-to-decimal precision loss by @Sigma-Ma in #2496
  • [AURON #2501] Support datediff function natively by @Sigma-Ma in #2502
  • [AURON #2499] perf: avoid identity take in GenerateExec by @lyne7-sc in #2500
  • [AURON #2497] Fix SortExec projected schema and sort key mapping by @lyne7-sc in #2498
  • [AURON #2351] Reduce native broadcast task serialization size by @Sigma-Ma in #2506
  • [AURON #2513] Optimize spill error handling and add an IOExceptiion for spill file missing by @Deegue in #2514
  • [AURON #2518] Cache maximum accumulator indices to avoid repeated scans by @lyne7-sc in #2519
  • [AURON #2509] Support date_add and date_sub functions natively by @Sigma-Ma in #2512
  • [AURON #2307] Add @transient to large fields and use NativePartition to reduce serialization overhead by @Deegue in #2515
  • [AURON #2511] Prune full-data-file Iceberg changelog delete tasks by @Sigma-Ma in #2523
  • [AURON #2530] Fix incorrect results in partial merge aggregation with sliced batches by @Deegue in #2531
  • [AURON #1743] Support ORC reader batch size by @Sigma-Ma in #2525
  • [AURON #2534] Fix native broadcast nested loop join diff by @Deegue in #2535
  • [AURON #2536] Add shared CPU join condition evaluation by @Deegue in #2537
  • [AURON #2529] Support add_months function natively by @Sigma-Ma in #2533
  • [AURON #2527] Populate identity partition values in native Iceberg scans by @lyne7-sc in #2528
  • [AURON #2524] Support unix_date and date_from_unix_date natively by @Sigma-Ma in #2526
  • [AURON #2521] Fix NULL predicate handling in aggregate filters by @lyne7-sc in #2522
  • [AURON #2538] Support and extend residual conditions in native sort-merge joins by @Deegue in #2543
  • [AURON #2547] Use TLP dist repositories in release script by @richox in #2548
  • [AURON #2549] update GitHub repo and README by @diliulou in #2549
  • [AURON #2550] docs: remove Incubating references after graduation by @diliulou in #2550
  • [AURON #2539] Support residual conditions in native broadcast hash joins by @Deegue in #2551
  • [AURON #2540] Support residual conditions in native shuffled hash joins by @Deegue in #2552
  • [AURON #2541] Support residual conditions in native broadcast nested loop joins by @Deegue in #2553
  • [AURON #2554] Support shared native memory manager and spill coordinator by @Deegue in #2556
  • [AURON #2544] Support next_day function natively by @Sigma-Ma in #2546
  • [AURON #2510] Add non-UTC timestamp coverage for native datediff by @Sigma-Ma in #2520
  • [AURON #2555] Support coordinated spilling for native host memory by @Deegue in #2557
  • [AURON #2559] Add native conversion diagnostics to Auron SQL UI. by @slfan1989 in #2560
  • [AURON #2562] Document native conversion diagnostics in the Spark UI. by @slfan1989 in #2563
  • [AURON #2567] Honor Spark memory overhead factors when calculating the native budget by @robreeves in #2568
  • [AURON #2118] make native environment initialization retriable after failure by @XorSum in #2151
  • [AURON #2564] Fix union input partition mapping during plan rewrites by @Deegue in #2565
  • [AURON #2577] Prevent stale Celeborn PID files from blocking CI startup by @lyne7-sc in #2578
  • [AURON #2569] Fix WindowGroupLimit fusion with multiple window expressions by @lyne7-sc in #2570
  • [AURON #2558] Support Iceberg _partition metadata in native scans by @Sigma-Ma in #2566
  • [AURON #2561] Support trunc function natively by @Sigma-Ma in #2571
  • [AURON #1919] Clean up Clippy code style/redundancy lints (Part 2) by @Sigma-Ma in #2585
  • [AURON #2579] Fix zero-column round-robin shuffle presort by @squanch-c in #2580
  • [AURON #2572] Support array_contains natively by @Sigma-Ma in #2573
  • [AURON #2593] Fix missing locality in NativeRDD by @Deegue in #2594
  • [AURON #2584] Implement native support for array_distinct by @Sigma-Ma in #2592
  • [AURON #2517] Align native float ordering with Spark by @Sigma-Ma in #2545
  • [AURON #2588] Preserve literal parameters during expression fallback by @lyne7-sc in #2589
  • [AURON #2608] Fix freezing of selected UDAF accumulator rows by @Deegue in #2616
  • [AURON #2609] Simplify native Parquet execution-plan output by @Deegue in #2617
  • [AURON #2610] Reuse LZ4 and ZSTD codec contexts across IPC blocks by @Deegue in #2618
  • [AURON #2581] Extract supported conjuncts for Iceberg changelog task … by @Sigma-Ma in #2583
  • [AURON #2611] Preserve unknown Parquet null counts by @Deegue in #2622
  • [AURON #2612] Optimize batch serialization with SIMD byte transposition by @Deegue in #2623
  • [AURON #2603] Fix SMJ fallback timing metrics under output backpressure by @lyne7-sc in #2604
  • [AURON #2596] Support decimal residual predicates in native Iceberg Parquet scan pruning by @Sigma-Ma in #2619
  • [AURON #2646] Hide the Auron tab in SHS when Auron is disabled by @robreeves in #2647
  • [AURON #2632] Fix shared row indices for aggregates with different FILTER predicates by @lyne7-sc in #2633
  • [AURON #2621] Restore Parquet row-group statistics pruning metrics by @Sigma-Ma in #2626
  • [AURON #2620] Support Spark's array_union natively by @Sigma-Ma in #2627
  • [AURON #2615] Separate explicit spills from coordinator admission by @Deegue in #2644
  • [AURON #2613] Support building the native engine with stable Rust by @Deegue in #2641
  • [AURON #2651] Support parallel reading for tables with small files by @richox in #2652
  • [AURON #2614] Add hybrid hash-join fallback for large build sides by @Deegue in #2643
  • [AURON #2653] Preserve native join row counts and simplify hybrid execution by @Deegue in #2655
  • [AURON #2654] Upgrade DataFusion to 55.2.0 and Arrow to 59.2.0 by @Deegue in #2656
  • [AURON #2664] Display available live SQL metrics in the Auron UI by @lyne7-sc in #2665
  • [AURON #2666] Synchronize batch evaluation in Spark UDF wrapper by @lihao712 in #2667
  • [AURON #2668] Fix xxhash64 with scalar arguments in datafusion-spark by @lihao712 in #2669
  • [AURON #2670] Evaluate coalesce natively in datafusion by @lihao712 in #2671
  • [AURON #2673] Default auron.cast.trimString to false by @lihao712 in #2674
  • [AURON #2687] Fix forceShuffledHashJoin functionality by @richox in #2688
  • [AURON #2681] Remove arrow shuffle files when a shuffle is unregistered by @lihao712 in #2682
  • [AURON #2685] Handle null scalar input in get_parsed_json_object by @lihao712 in #2686
  • [AURON #2678] Normalize sliced Arrow arrays before exporting to Java by @Deegue in #2694
  • [AURON #2679] Share batch coalescing between execution streams and IPC readers by @Deegue in #2693
  • [AURON #2662] Classify adaptive shuffle readers as wrappers in Auron UI by @robreeves in #2663
  • [AURON #2676] Add a switch for native datetime function conversion by @lihao712 in #2677
  • [AURON #2683] Return execution errors for aggregate placeholder evaluation by @Deegue in #2701
  • [AURON #2680] Reuse compatible datafusion-spark extension functions by @Deegue in #2700
  • [AURON #2672] Fix native JSON scalar batch handling by @squanch-c in #2692
  • [AURON #2629] Keep native Iceberg scans for unaffected projections after nested schema evolution by @Sigma-Ma in #2658
  • [AURON #2576] Support arrays_overlap natively by @Sigma-Ma in #2582
  • [AURON #2684] Return NaN for negative sqrt inputs by @lyne7-sc in #2697
  • [AURON #2635] Fix LZ4_HADOOP parquet framing interop with parquet-java by @COLDMANYU in #2638
  • [AURON #2659] Count distinct data files in native Iceberg snapshot scan metrics by @Sigma-Ma in #2696
  • [AURON #2705] Fix Clippy failure in arrays_overlap test by @Sigma-Ma in #2706
  • [AURON #2707] Align struct comparison with Spark for max/min aggregations by @richox in #2708
  • [AURON #2689] Use scan-range lengths when estimating native Iceberg partition sizes by @Sigma-Ma in #2699
  • [AURON #2716] Bump version to 9.0.0 by @weimingdiit in #2717
  • [AURON #2721] Exclude IDEA files from source release archives by @weimingdiit in #2722

New Contributors

  • @zhaohehuhu made their first contribution in #2374
  • @goutamadwant made their first contribution in #2456
  • @squanch-c made their first contribution in #2580
  • @COLDMANYU made their first contribution in #2638

Contributors

Thanks to all contributors who made this release possible:

@COLDMANYU, @cxzl25, @Deegue, @dependabot[bot], @diliulou, @Flyangz, @goutamadwant, @lihao712, @lyne7-sc, @richox, @robreeves, @ShreyeshArangath, @Sigma-Ma, @slfan1989, @squanch-c, @Tartarus0zm, @weimingdiit, @weiqingy, @wForget, @XorSum, @yew1eb, @zhaohehuhu, @zuston

Reviewers

Thanks to everyone who reviewed pull requests for this release:

@cxzl25, @diliulou, @lihao2077tech, @lihao712, @lyne7-sc, @merrily01, @richox, @robreeves, @ShreyeshArangath, @slfan1989, @SteNicholas, @Tartarus0zm, @weimingdiit, @wForget, @yew1eb

Full Changelog

Full Changelog: v8.0.0...v9.0.0

Download

VersionDateSourceChecksumSignature
9.0.0Nov 19 2026apache-auron-9.0.0-source.tgzsha512asc

The release artifacts can be verified using the Apache Auron KEYS file. See the Apache Software Foundation's release verification instructions for details.

See the corresponding GitHub Release for the release notes and source tag.

Next
v8.0.0-incubating