Apache Auron 8.0.0-incubating

Apache Auron 8.0.0-incubating is a new Apache Incubator release. This release significantly expands Flink integration, native scan support for lakehouse table formats, native expression and function coverage, build and CI compatibility, and runtime reliability.

New Features

Flink integration: Introduced the Flink Auron adaptor, configuration framework, expression converter framework, native Calc operators, and the structural Flink assembly deployment JAR.
Native Kafka source for Flink: Added a native Kafka table source and scan execution with checkpointing, watermark generation, idleness handling, partition discovery, JSON format support, and Flink metrics integration.
Lakehouse native scans: Added and expanded native scan support for Apache Iceberg, Apache Hudi, and Apache Paimon, including Copy-on-Write tables, Iceberg changelog scans, metadata columns, residual filters, scan metrics, and Paimon DataSource V2 scans.
Native expressions and functions: Added native support for aggregate FILTER (WHERE ...) clauses, Cast, comparison, logical, and mathematical operators, along with additional array, map, date, string, and unary expressions.
Window functions and joins: Added native implementations of lead, nth_value, percent_rank, and cume_dist, plus residual join-condition support for SortMergeJoin and ShuffledHashJoin.
Native serialization: Introduced native JSON and Protocol Buffers deserializers and expanded Arrow-to-Flink conversion support for basic, temporal, and composite types.

Improvements

Correctness testing: Expanded test scaffolding and suite wrappers across supported Spark versions for expressions, functions, aggregates, sorting, and Parquet behavior.
Spark compatibility: Added Spark 4.0 and 4.1 support to build and development workflows and fixed Scala 2.13 CI compatibility.
Build and packaging: Added macOS build support, Maven thread controls, improved command-line option parsing, standardized Cargo workspace settings, and refined release artifact naming.
Flink behavior: Improved configuration naming, metrics reporting, watermark handling, partition discovery, adaptor lifecycle management, and deployment packaging.
Native scan observability: Added conversion-reason reporting and exposed file and partition metrics for supported table-format scans.
Dependency maintenance: Updated Maven, Hadoop, Celeborn, Kafka, Tokio, Prost, Chrono, Log, and other build/runtime dependencies.

Bug Fixes

Fixed Kafka source cases that emitted no records or failed to generate watermarks.
Fixed native Paimon filtering and metadata-column correctness issues.
Fixed Iceberg partitioning, schema evolution, metadata-column, and residual-filter behavior.
Fixed Hudi scan option handling and prevented unsupported incremental queries from being converted to native scans.
Fixed native engine panics, core dumps, memory leaks, closed-channel crashes, and Java thread interruption errors.
Fixed driver-side spill NPEs, Uniffle reader and writer synchronization issues, and History Server compatibility when build information is missing.
Fixed timestamp timezone handling, ORC range bounds, stale build-version values, Scala 2.13 CI errors, and native metrics retrieval.

Notable Changes

Flink becomes a first-class integration: Auron 8.0.0 introduces a broader Flink runtime and planning foundation, including native Kafka ingestion and Calc conversion.
Broader lakehouse acceleration: Native scan coverage now spans key Apache Iceberg, Apache Hudi, and Apache Paimon scenarios with improved correctness and observability.
Expanded native execution: More Spark and Flink operators, scalar functions, aggregate filters, window functions, and join conditions can execute natively.
Stronger cross-version validation: Correctness suites and build workflows now cover more Spark, Scala, JDK, operating-system, and architecture combinations.
Improved runtime stability: This release resolves multiple crash, panic, NPE, thread-safety, watermark, and scan-correctness problems.

What's Changed

Flink integration and Kafka source

Fix duplicate initialization of the FlinkAuronConfiguration object by @Tartarus0zm in #2357
Fix Auron Kafka source emitting no records without a watermark by @weiqingy in #2326
Make auron-flink-assembly the structural deployment jar by @weiqingy in #2292
Convert Flink Calc operators to Native Calc operators by @weiqingy in #2283
Introduce FlinkAuronCalcOperator by @weiqingy in #2263
Fix KafkaScanExec error when using the JSON format by @Tartarus0zm in #2267
Bump org.apache.kafka:kafka-clients from 3.9.1 to 3.9.2 in /auron-flink-extension/auron-flink-runtime by @dependabot[bot] in #2198
Integration with Flink metrics in AuronKafkaSourceFunction by @xuzifu666 in #2186
Introduce Flink expression-level converter framework by @weiqingy in #2146
Kafka source support watermark with idleness by @Tartarus0zm in #2142
Make libInputStream auto-close when creating in Flink/Spark AuronAdaptor. by @RocMarshal in #2117
Optimize option names to align with the Flink community by @Tartarus0zm in #2113
Support kafka partition discovery by @Tartarus0zm in #2111
Introduce kafka mock source by @Tartarus0zm in #2106
Fix AuronKafkaSourceFunction watermark was not generated by @Tartarus0zm in #2098
Bump org.apache.kafka:kafka-clients from 3.4.0 to 3.9.1 in /auron-flink-extension/auron-flink-runtime by @dependabot[bot] in #2092
Add ArrowFieldWriters for temporal and composite types by @x-tong in #2086
AuronKafkaSourceFunction support generating watermarks by @Tartarus0zm in #2089
AuronKafkaSourceFunction support flink checkpoint by @Tartarus0zm in #2084
Introduce Auron Flink Kafka TableSource by @Tartarus0zm in #2082
Add ArrowFieldWriter and FlinkArrowWriter for basic types by @x-tong in #2079
Introduce Arrow to Flink RowData reader by @weiqingy in #2063
Introduce Auron Native Kafka DynamicTableFactory by @Tartarus0zm in #2078
Introduce KafkaScanExec by @Tartarus0zm in #2072
Introduce Flink Auron interface by @Tartarus0zm in #2057
Introduce FlinkAuronAdaptor and FlinkAuronAdaptorProvider by @Tartarus0zm in #2056
Introduce FlinkAuronConfiguration by @Tartarus0zm in #2055

Native scans and table-format support

Support Iceberg column rename and drop-then-add in the native scan by @lyne7-sc in #2322
fix: Handle Paimon metadata columns in V2 native scan by @lyne7-sc in #2367
Support insert-only Iceberg changelog native scan by @weimingdiit in #2254
fix: Native Paimon V2 scan returns wrong results with native filter by @lyne7-sc in #2365
Support native scan for Paimon DSv2 (BatchScanExec) COW tables by @zhuxiangyi in #2313
Fix NativeIcebergTableScan to respect SinglePartition outputPartitioning by @lyne7-sc in #2342
Add support for auron.never.convert.reason in Hudi scan scenarios by @guixiaowen in #2293
Add parquet suite wrappers for Spark versions by @myandpr in #2281
Fallback Hudi incremental queries from native scan conversion by @weimingdiit in #2274
Implement native support for ORC InsertIntoHiveTable writes by @weimingdiit in #2191
Add support for auron.never.convert.reason in Iceberg scan scenarios by @guixiaowen in #2237
Add native Hudi scan coverage for partitioned COW tables by @weimingdiit in #2251
Support native Hudi scan for MOR read-optimized queries by @weimingdiit in #2249
Reuse Hudi table properties when detecting native scan support by @weimingdiit in #2247
Support Iceberg _spec_id metadata in native scan by @weimingdiit in #2218
Expose numFiles and numPartitions metrics for native Iceberg scan by @weimingdiit in #2220
Handle Hudi scan options case-insensitively by @weimingdiit in #2235
Remove hard-coded Iceberg scan class name detection using type-based check by @guixiaowen in #2226
Add native support for the _file metadata column by @weimingdiit in #2184
Support native Iceberg scans with residual filters via scan pruning and post-scan native filter by @weimingdiit in #2164
Add Native Scan Support for Apache Hudi Copy-On-Write Tables. by @slfan1989 in #2031
chore: add Iceberg version to AuronBuildInfo by @XorSum in #2095
Fix ORC range start index out of range for slice of length by @cxzl25 in #1986
Add Native Scan Support for Apache Iceberg Copy-On-Write Tables. by @slfan1989 in #2016

Native expressions, operators, and functions

Detect unexpected fallback in expression tests by @weimingdiit in #2332
Support native flatten for mixed child array nullability by @weimingdiit in #2344
Support more literal patterns in native split by @weimingdiit in #2336
Support array input for native reverse by @weimingdiit in #2338
Refactor and rename cast test suites for ANSI mode on and off by @cxzl25 in #2324
Add expression test wrappers for supported Spark versions by @peter941221 in #2284
support SQL aggregate FILTER (WHERE ...) clause in native execution by @yew1eb in #2290
Add end-to-end IT coverage for composed native Calc predicates and expressions by @weiqingy in #2306
Convert Cast operators to native operators by @weiqingy in #2303
Convert comparison operators and LIKE to native operators by @weiqingy in #2298
Convert logical operators to Native operators by @weiqingy in #2297
Support binary input for Spark substring function by @lyne7-sc in #2262
Implement native function of make_date by @guixiaowen in #2241
Implement native function of str_to_map by @weimingdiit in #2190
Convert math operators to Auron native operators by @weiqingy in #2167
Implement native function of map_from_entries by @weimingdiit in #2169
Implement native function of map_from_arrays by @weimingdiit in #2165
Implement native function of months_between by @weimingdiit in #2149
Implement native function of weekofyear by @weimingdiit in #2131
Implement native function of map_concat by @weimingdiit in #2141
Add native support for acosh function by @weimingdiit in #2135
Implement native function of dayofweek by @weimingdiit in #2129
Support UnaryMinus expression by @yew1eb in #2051

Window functions and joins

Implement native support for inner residual join conditions on SMJ/SHJ by @weimingdiit in #2197
Implement native support for cume_dist window function by @weimingdiit in #2205
Implement native support for percent_rank window function by @weimingdiit in #2204
Implement native support for nth_value window function by @weimingdiit in #2203
Implement native support for lead window function by @weimingdiit in #2188

Correctness testing and Spark-version coverage

Add sort suite wrappers for Spark versions by @myandpr in #2275
Add aggregate suite wrappers for Spark versions by @myandpr in #2256
[Correctness Testing] Add function-related tests by @lyne7-sc in #2245
correctness testing: setup correctness testing scaffolding for all spark versions by @ShreyeshArangath in #2209
Fix rust lint test fail by @Tartarus0zm in #2100
skipping the plan stability validation for tpcds q54 test in spark3.5 by @XorSum in #2071

Build, CI, packaging, and dependencies

Bump actions/cache from 5 to 6 by @dependabot[bot] in #2353
Bump actions/checkout from 6 to 7 by @dependabot[bot] in #2345
Bump itertools from 0.14.0 to 0.15.0 by @dependabot[bot] in #2340
Fix master build: pass keys to getDefaultNativeMetrics in NativeOrcInsertIntoHiveTableBase by @weiqingy in #2334
Bump prost from 0.14.3 to 0.14.4 by @dependabot[bot] in #2314
Bump log from 0.4.31 to 0.4.32 by @dependabot[bot] in #2312
Bump chrono from 0.4.44 to 0.4.45 by @dependabot[bot] in #2311
Bump log from 0.4.30 to 0.4.31 by @dependabot[bot] in #2305
bump maven from 3.9.12 to 3.9.16 by @cxzl25 in #2300
Bump log from 0.4.29 to 0.4.30 by @dependabot[bot] in #2288
Bump prost-reflect from 0.16.3 to 0.16.4 by @dependabot[bot] in #2286
Add support for Spark 4.0 and 4.1 in build workflows by @cxzl25 in #2261
[INFRA] Set up default rulesets for default and release branches by @asf-gitbox-commits in #2271
[build] Fix stale auronver in build-macos-releases.yml by @yew1eb in #2278
Bump rkyv from 0.8.11 to 0.8.16 by @dependabot[bot] in #2270
[CI] Fix macOS build by disabling cargo bin caching by @cxzl25 in #2269
Bump lz4_flex from 0.13.0 to 0.13.1 by @dependabot[bot] in #2259
Bump tokio from 1.52.2 to 1.52.3 by @dependabot[bot] in #2258
Bump tokio from 1.52.1 to 1.52.2 by @dependabot[bot] in #2233
auron-build.sh fails to parse options after -D arguments by @guixiaowen in #2228
[BUILD] Add -Xlint:_ to scala-maven-plugin and resolve related warnings by @1fanwang in #2216
Add MacOS build by @cxzl25 in #1839
Bump rustls-webpki from 0.103.12 to 0.103.13 by @dependabot[bot] in #2214
Extract common package configs to workspace Cargo.toml by @yew1eb in #2004
Bump Celeborn 0.6.3 by @myandpr in #2212
Bump rustls-webpki from 0.103.4 to 0.103.12 by @dependabot[bot] in #2207
Bump tokio from 1.52.0 to 1.52.1 by @dependabot[bot] in #2210
Bump tokio from 1.51.1 to 1.52.0 by @dependabot[bot] in #2206
Bump rand from 0.9.2 to 0.9.3 by @dependabot[bot] in #2201
Bump sonic-rs from 0.5.7 to 0.5.8 by @dependabot[bot] in #2114
Bump tokio from 1.51.0 to 1.51.1 by @dependabot[bot] in #2187
Bump tokio from 1.50.0 to 1.51.0 by @dependabot[bot] in #2168
Standardize Cargo workspace edition and resolver configuration. by @slfan1989 in #1997
Remove pull_request_template.md comment by @XorSum in #2136
Update artifact name to include Celeborn and Uniffle versions by @cxzl25 in #2122
Bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #2109
Clean up rust default lints by @yew1eb in #2039
Bump lz4_flex from 0.12.0 to 0.13.0 by @dependabot[bot] in #2091
Bump once_cell from 1.21.3 to 1.21.4 by @dependabot[bot] in #2088
Update copyright year in NOTICE file by @CalvinKirs in #2087
Bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #2075
Bump tokio from 1.49.0 to 1.50.0 by @dependabot[bot] in #2070
[BUILD] Add --threads option to control Maven build parallelism. by @slfan1989 in #2021
Bump futures from 0.3.31 to 0.3.32 by @dependabot[bot] in #2065
Bump bytes from 1.11.0 to 1.11.1 by @dependabot[bot] in #1983
History Server fails when BuildInfo event is missing. by @slfan1989 in #2012
Bump futures-util from 0.3.31 to 0.3.32 by @dependabot[bot] in #2010
Bump hadoop from 3.4.2 to 3.4.3 by @yew1eb in #2044
Bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #2041
Bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #2040
[BUILD] Add Spark 4.x support to dev/reformat script. by @slfan1989 in #2018
Fix Scala 2.13 CI NoClassDefFoundError by @cxzl25 in #2048
Bump sonic-rs from 0.5.6 to 0.5.7 by @dependabot[bot] in #2038
Bump chrono from 0.4.43 to 0.4.44 by @dependabot[bot] in #2036

Reliability, bug fixes, and cleanup

Honor config alt keys when reading from SQLConf by @robreeves in #2361
chore: enable copilot code review on push by @cxzl25 in #2355
Optimize native metrics retrieval by passing keys directly by @lsm1 in #1982
Upgrade hashbrown to 0.17.1 and adapt API by @cxzl25 in #2302
Support compile with Celeborn 0.6 and spark 4.0 by @ftong2020 in #2243
Date-part extraction functions missing timezone handling for Timestamp inputs by @ShreyeshArangath in #2156
Wire existing native scalar functions in NativeConverters by @ShreyeshArangath in #2162
Fix typos and unnecessary memory allocation in native engine by @Deegue in #2077
fix: change const to static in batch_size() to fix OnceCell caching by @yew1eb in #2159
Set thread name for ArrowFFIExporter by @cxzl25 in #2148
Native thread name appends the original thread name by @cxzl25 in #2144
fix pb list name missing by @Tartarus0zm in #2143
Fix some typos in native by @XorSum in #2139
Add isTaskRunning method to check task status in SparkAuronAdaptor by @cxzl25 in #2120
Introduce native json deserializer by @Tartarus0zm in #2112
Fix driver spill NPE error by @Deegue in #2108
Migrate the Native dependency configuration from SparkAuronConfiguration to AuronConfiguration by @Tartarus0zm in #2105
Initializing JavaClasses in JNI and decoupling Spark by @Tartarus0zm in #2103
Fix panic/coredumps/memory leaks and Java thread InterruptedException errors. by @BrytonLee in #1980
Introduce native protobuf deserializer by @Tartarus0zm in #2069
NPE fix in AuronUniffleShuffleReader by @slfan1989 in #2053
PR title check by @XorSum in #2066
Migrate from deprecated JavaConverters to CollectionConverters. by @slfan1989 in #2027
[BUG] Native engine panic on closed channel causes JVM crash. by @slfan1989 in #2023
Fix extra brace in SparkOnHeapSpillManager log message. by @slfan1989 in #2029
Fix thread-safety issues in UnifflePartitionWriter synchronization. by @slfan1989 in #2033

Contributors

Thanks to all contributors who made this release possible:

bkhan, Bryton Lee, Calvin Kirs, cxzl25, dependabot[bot], ftong2020, guixiaowen, linfeng, LSM, Ming Wei, Peter Chen J., Rob Reeves, Shreyesh, slfan1989, Stefan Wang, The Apache Software Foundation, Weiqing Yang, Xiangyi Zhu, xTong, xuzifu666, yaommen, yew1eb, Yizhong Zhang, Yuepeng Pan, zhangmang

Version	Date	Source	Checksum	Signature
8.0.0-incubating	Jul 18 2026	apache-auron-8.0.0-incubating-source.tgz	sha512	asc

The release artifacts can be verified using the Apache Auron KEYS file. See the Apache Software Foundation's release verification instructions for details.