New Features

  • New Configurations: Introduced settings for decimal operations, JSON parsing fallback, Parquet reader, native logging, and compression.
  • Memory Management: Improve memory management using Linux RSS (resident set size).
  • Operators: Supports operator fusion in Sort -> SortMergeJoin execution, reducing costs of join key serialization.
  • Enhanced Compatibility: Added support for JDK 17 and Scala 2.13.
  • New Functions: Added support for trim in casts and extended hashing function coverage.

Improvements

  • Stability: Improved handling of stage retry on shuffle failures and memory spilling.
  • Modularity: Restructured codebase by extracting Celeborn, Uniffle, and Paimon into separate 3rdparty modules.
  • Observability: Improved logging with Thread IDs and enhanced Spark UI metrics for skew detection.
  • Uniffle Integration: Improved support and documentation for Uniffle shuffle manager.
  • Minor Performance Improvement: Optimized batch serde, array interleavig and coalescing.
  • Build & CI: Enhanced build scripts, added ARM support, and streamlined the CI process.

Bug Fixes

  • Data Correctness: Fixed critical issues in join logic, value comparisons, and hash calculations.
  • Memory Leaks & Crashes: Resolved memory management issues and NPEs.
  • Execution Engine: Fixed errors in outer generate, UDTF execution, and Parquet sink tasks.
  • Integration: Corrected issues with 3rdparty systems like Celeborn and Uniffle.

NOTE: This release includes a significant number of performance optimizations, memory management improvements, bug fixes, and new features, with notable enhancements in shuffle management, execution engine optimization, and third-party integration. Some minor changes are not included in the above list, please see the commit list for more details.

New Contributors

  • @diliulou made their first contribution in https://github.com/apache/auron/pull/1023
  • @eden123456789 made their first contribution in https://github.com/apache/auron/pull/1078
  • @turboFei made their first contribution in https://github.com/apache/auron/pull/1090
  • @xuzifu666 made their first contribution in https://github.com/apache/auron/pull/1278
  • @FMX made their first contribution in https://github.com/apache/auron/pull/1298
  • @hhhizzz made their first contribution in https://github.com/apache/auron/pull/1313

Full Changelog: https://github.com/apache/auron/compare/v5.0.0...v6.0.0

Download

VersionDateSource
6.0.0Oct 17 2025v6.0.0.zip
v6.0.0.tar.gz