Release DataFusion 52.1.0 or 52.0.1 (minor/patch) Release (Jan 2026)

[alamb]

Is your feature request related to a problem or challenge?

• Part of [EPIC] DataFusion Releases #19783
• Follow on to Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) #18566

Let's plan a 52.0.1 or 52.1.0 patch release to fix any bugs / issues found in 52.0.0. @bert-beyondloops found one in #19781

Steps:

• Update version and changelog
• Test with DataFusion Python
• Test with DataFusion Comet
• Test with delta.rs:
• Test vortex:
• Test with iceberg-rust:
• Test with LakeSail:
• Test with parquet viewer
• Test with datafusion-materialized-views
• Voting Thread:
• Publish to crates.io

Potential Content:

Bug Fixes

• Internal error: Assertion failed: !self.finished: LimitedBatchCoalescer: cannot push batch after finish #19781 (thanks @bert-beyondloops)
• fix: expose ListFilesEntry #19804
• Different results in TPC-DS q14 depending on the number of partitions #19849

Minor Features:

• TBD

[lonless9]

I hope this PR can be included in this minor release!

• fix: expose ListFilesEntry #19804

[alamb]

I hope this PR can be included in this minor release!

• fix: expose ListFilesEntry #19804

Sure @lonless9 -- that looks like a great idea.

Can you make a backport PR to branch-52?

Here is an example PR: #19705

(basically just git cherry-pick the commit from main to the branch

[gabotechs]

I'm seeing something weird in TPC-DS Q14, something that the tests in https://github.com/datafusion-contrib/datafusion-distributed/ seem to be catching, originally reported here pydantic/datafusion-distributed#2 (comment).

It seems like the returned results for TPC-DS Q14 are different from version 51 to 52:

Result of pretty_format_batches for TPC-DS Q14 in v51

+---------+------------+------------+---------------+--------------------+------------------+
| channel | i_brand_id | i_class_id | i_category_id | sum_sales          | sum_number_sales |
+---------+------------+------------+---------------+--------------------+------------------+
|         |            |            |               | 674173362.5100003  | 155629           |
| catalog |            |            |               | 237410857.47000003 | 46322            |
| catalog | 1001001.0  |            |               | 1697729.02         | 347              |
| catalog | 1001001.0  | 1.0        |               | 855204.24          | 167              |
| catalog | 1001001.0  | 1.0        | 1.0           | 115019.61000000002 | 20               |
| catalog | 1001001.0  | 1.0        | 2.0           | 146344.46999999997 | 27               |
| catalog | 1001001.0  | 1.0        | 3.0           | 22597.190000000002 | 3                |
| catalog | 1001001.0  | 1.0        | 4.0           | 107555.42999999998 | 23               |
| catalog | 1001001.0  | 1.0        | 5.0           | 122521.31000000001 | 25               |
| catalog | 1001001.0  | 1.0        | 6.0           | 16883.97           | 3                |
| catalog | 1001001.0  | 1.0        | 7.0           | 46329.78           | 9                |
| catalog | 1001001.0  | 1.0        | 8.0           | 77861.85           | 13               |
| catalog | 1001001.0  | 1.0        | 9.0           | 99985.34999999999  | 21               |
| catalog | 1001001.0  | 1.0        | 10.0          | 100105.28000000001 | 23               |
| catalog | 1001001.0  | 2.0        |               | 125167.21999999997 | 24               |
| catalog | 1001001.0  | 2.0        | 2.0           | 43967.969999999994 | 7                |
| catalog | 1001001.0  | 2.0        | 3.0           | 68565.37999999999  | 14               |
| catalog | 1001001.0  | 2.0        | 5.0           | 12633.87           | 3                |
| catalog | 1001001.0  | 3.0        |               | 198685.08000000002 | 43               |
| catalog | 1001001.0  | 3.0        | 1.0           | 11100.79           | 5                |
| catalog | 1001001.0  | 3.0        | 2.0           | 60551.64           | 14               |
| catalog | 1001001.0  | 3.0        | 4.0           | 28455.230000000003 | 4                |
| catalog | 1001001.0  | 3.0        | 6.0           | 36821.61           | 7                |
| catalog | 1001001.0  | 3.0        | 7.0           | 17250.82           | 6                |
| catalog | 1001001.0  | 3.0        | 8.0           | 14426.920000000002 | 4                |
| catalog | 1001001.0  | 3.0        | 9.0           | 30078.07           | 3                |
| catalog | 1001001.0  | 4.0        |               | 109585.96999999999 | 31               |
| catalog | 1001001.0  | 4.0        | 2.0           | 45473.85           | 13               |
| catalog | 1001001.0  | 4.0        | 3.0           | 16558.92           | 8                |
| catalog | 1001001.0  | 4.0        | 4.0           | 47553.2            | 10               |
| catalog | 1001001.0  | 5.0        |               | 59790.61           | 17               |
| catalog | 1001001.0  | 5.0        | 9.0           | 30112.109999999997 | 12               |
| catalog | 1001001.0  | 5.0        | 10.0          | 29678.5            | 5                |
| catalog | 1001001.0  | 6.0        |               | 10261.82           | 3                |
| catalog | 1001001.0  | 6.0        | 9.0           | 10261.82           | 3                |
| catalog | 1001001.0  | 7.0        |               | 18244.94           | 3                |
| catalog | 1001001.0  | 7.0        | 7.0           | 18244.94           | 3                |
| catalog | 1001001.0  | 8.0        |               | 55768.46           | 13               |
| catalog | 1001001.0  | 8.0        | 7.0           | 28872.49           | 7                |
| catalog | 1001001.0  | 8.0        | 10.0          | 26895.969999999998 | 6                |
| catalog | 1001001.0  | 9.0        |               | 30944.190000000002 | 5                |
| catalog | 1001001.0  | 9.0        | 6.0           | 30944.190000000002 | 5                |
| catalog | 1001001.0  | 11.0       |               | 82810.87           | 12               |
| catalog | 1001001.0  | 11.0       | 9.0           | 82810.87           | 12               |
| catalog | 1001001.0  | 12.0       |               | 38427.520000000004 | 9                |
| catalog | 1001001.0  | 12.0       | 10.0          | 38427.520000000004 | 9                |
| catalog | 1001001.0  | 15.0       |               | 112838.1           | 20               |
| catalog | 1001001.0  | 15.0       | 9.0           | 53508.79000000001  | 7                |
| catalog | 1001001.0  | 15.0       | 10.0          | 59329.31           | 13               |
| catalog | 1001002.0  |            |               | 3527831.3300000005 | 706              |
| catalog | 1001002.0  | 1.0        |               | 2673969.8900000006 | 530              |
| catalog | 1001002.0  | 1.0        | 1.0           | 2673969.8900000006 | 530              |
| catalog | 1001002.0  | 2.0        |               | 140831.91000000003 | 29               |
| catalog | 1001002.0  | 2.0        | 1.0           | 140831.91000000003 | 29               |
| catalog | 1001002.0  | 3.0        |               | 320175.86999999994 | 67               |
| catalog | 1001002.0  | 3.0        | 1.0           | 320175.86999999994 | 67               |
| catalog | 1001002.0  | 4.0        |               | 133287.96000000002 | 21               |
| catalog | 1001002.0  | 4.0        | 1.0           | 133287.96000000002 | 21               |
| catalog | 1001002.0  | 5.0        |               | 16606.9            | 9                |
| catalog | 1001002.0  | 5.0        | 1.0           | 16606.9            | 9                |
| catalog | 1001002.0  | 6.0        |               | 15133.01           | 4                |
| catalog | 1001002.0  | 6.0        | 1.0           | 15133.01           | 4                |
| catalog | 1001002.0  | 7.0        |               | 24471.260000000002 | 10               |
| catalog | 1001002.0  | 7.0        | 1.0           | 24471.260000000002 | 10               |
| catalog | 1001002.0  | 8.0        |               | 63773.049999999996 | 12               |
| catalog | 1001002.0  | 8.0        | 1.0           | 63773.049999999996 | 12               |
| catalog | 1001002.0  | 9.0        |               | 9167.189999999999  | 3                |
| catalog | 1001002.0  | 9.0        | 1.0           | 9167.189999999999  | 3                |
| catalog | 1001002.0  | 12.0       |               | 29108.420000000002 | 7                |
| catalog | 1001002.0  | 12.0       | 1.0           | 29108.420000000002 | 7                |
| catalog | 1001002.0  | 15.0       |               | 31143.449999999997 | 6                |
| catalog | 1001002.0  | 15.0       | 1.0           | 31143.449999999997 | 6                |
| catalog | 1001002.0  | 16.0       |               | 70162.42           | 8                |
| catalog | 1001002.0  | 16.0       | 1.0           | 70162.42           | 8                |
| catalog | 1002001.0  |            |               | 2114110.72         | 380              |
| catalog | 1002001.0  | 1.0        |               | 348693.97          | 55               |
| catalog | 1002001.0  | 1.0        | 1.0           | 76392.12999999999  | 14               |
| catalog | 1002001.0  | 1.0        | 2.0           | 118394.33          | 21               |
| catalog | 1002001.0  | 1.0        | 4.0           | 29395.79           | 5                |
| catalog | 1002001.0  | 1.0        | 5.0           | 35541.97           | 4                |
| catalog | 1002001.0  | 1.0        | 6.0           | 26104.36           | 3                |
| catalog | 1002001.0  | 1.0        | 9.0           | 18793.969999999998 | 4                |
| catalog | 1002001.0  | 1.0        | 10.0          | 44071.42           | 4                |
| catalog | 1002001.0  | 2.0        |               | 1233961.6999999997 | 225              |
| catalog | 1002001.0  | 2.0        | 1.0           | 239511.02000000002 | 51               |
| catalog | 1002001.0  | 2.0        | 2.0           | 147993.14          | 26               |
| catalog | 1002001.0  | 2.0        | 3.0           | 100086.92999999998 | 17               |
| catalog | 1002001.0  | 2.0        | 4.0           | 53524.41999999999  | 13               |
| catalog | 1002001.0  | 2.0        | 5.0           | 48494.06           | 10               |
| catalog | 1002001.0  | 2.0        | 6.0           | 142857.03999999998 | 20               |
| catalog | 1002001.0  | 2.0        | 7.0           | 116557.98          | 16               |
| catalog | 1002001.0  | 2.0        | 8.0           | 92743.92999999998  | 24               |
| catalog | 1002001.0  | 2.0        | 9.0           | 203943.98999999996 | 38               |
| catalog | 1002001.0  | 2.0        | 10.0          | 88249.19           | 10               |
| catalog | 1002001.0  | 3.0        |               | 91054.32           | 17               |
| catalog | 1002001.0  | 3.0        | 2.0           | 25171.130000000005 | 6                |
| catalog | 1002001.0  | 3.0        | 7.0           | 27766.7            | 3                |
| catalog | 1002001.0  | 3.0        | 8.0           | 38116.49           | 8                |
| catalog | 1002001.0  | 4.0        |               | 182427.69          | 32               |
| catalog | 1002001.0  | 4.0        | 1.0           | 66896.68000000001  | 15               |
+---------+------------+------------+---------------+--------------------+------------------+

Result of pretty_format_batches for TPC-DS Q14 in v52

+---------+------------+------------+---------------+--------------------+------------------+
| channel | i_brand_id | i_class_id | i_category_id | sum_sales          | sum_number_sales |
+---------+------------+------------+---------------+--------------------+------------------+
|         |            |            |               | 47788579.86999994  | 11068            |
|         |            |            |               | 46294358.79000003  | 10609            |
|         |            |            |               | 40499040.27        | 9321             |
|         |            |            |               | 37952602.75000005  | 8889             |
|         |            |            |               | 50256292.020000026 | 11540            |
|         |            |            |               | 27943616.979999993 | 6397             |
|         |            |            |               | 43114338.769999966 | 10000            |
|         |            |            |               | 56239021.03999995  | 13003            |
|         |            |            |               | 25682800.660000004 | 6012             |
|         |            |            |               | 38529122.80999996  | 8922             |
|         |            |            |               | 59222982.15999998  | 13528            |
|         |            |            |               | 48322926.85999996  | 11228            |
|         |            |            |               | 39166012.100000024 | 9010             |
|         |            |            |               | 32661391.260000017 | 7453             |
|         |            |            |               | 43315152.09999997  | 10008            |
|         |            |            |               | 37185124.06999998  | 8641             |
| catalog |            |            |               | 16671923.720000004 | 3228             |
| catalog |            |            |               | 16630833.01        | 3143             |
| catalog |            |            |               | 14038550.020000003 | 2798             |
| catalog |            |            |               | 13135427.840000002 | 2638             |
| catalog |            |            |               | 17604907.44        | 3399             |
| catalog |            |            |               | 10119873.490000004 | 1959             |
| catalog |            |            |               | 14698922.719999986 | 2919             |
| catalog |            |            |               | 19534422.180000022 | 3931             |
| catalog |            |            |               | 9075046.950000005  | 1756             |
| catalog |            |            |               | 13829338.200000005 | 2662             |
| catalog |            |            |               | 21769645.880000025 | 4087             |
| catalog |            |            |               | 16890254.589999996 | 3343             |
| catalog |            |            |               | 13897305.679999998 | 2680             |
| catalog |            |            |               | 11719010.149999999 | 2217             |
| catalog |            |            |               | 14773719.710000006 | 2947             |
| catalog |            |            |               | 13021675.88999999  | 2615             |
| catalog | 1001001.0  |            |               | 188446.33000000002 | 41               |
| catalog | 1001001.0  |            |               | 53508.79           | 7                |
| catalog | 1001001.0  |            |               | 100105.28          | 23               |
| catalog | 1001001.0  |            |               | 114412.26999999999 | 25               |
| catalog | 1001001.0  |            |               | 77231.70000000001  | 15               |
| catalog | 1001001.0  |            |               | 174489.15000000002 | 42               |
| catalog | 1001001.0  |            |               | 206490.30000000002 | 38               |
| catalog | 1001001.0  |            |               | 45473.85           | 13               |
| catalog | 1001001.0  |            |               | 146344.47          | 27               |
| catalog | 1001001.0  |            |               | 152599.38          | 28               |
| catalog | 1001001.0  |            |               | 206412.37          | 36               |
| catalog | 1001001.0  |            |               | 119368.21          | 23               |
| catalog | 1001001.0  |            |               | 45014.149999999994 | 12               |
| catalog | 1001001.0  |            |               | 50948.8            | 14               |
| catalog | 1001001.0  |            |               | 16883.97           | 3                |
| catalog | 1001001.0  | 1.0        |               | 100105.28          | 23               |
| catalog | 1001001.0  | 1.0        |               | 99985.34999999999  | 21               |
| catalog | 1001001.0  | 1.0        |               | 107555.43000000001 | 23               |
| catalog | 1001001.0  | 1.0        |               | 161349.39          | 29               |
| catalog | 1001001.0  | 1.0        |               | 146344.47          | 27               |
| catalog | 1001001.0  | 1.0        |               | 122521.31000000001 | 25               |
| catalog | 1001001.0  | 1.0        |               | 77861.85           | 13               |
| catalog | 1001001.0  | 1.0        |               | 22597.190000000002 | 3                |
| catalog | 1001001.0  | 1.0        |               | 16883.97           | 3                |
| catalog | 1001001.0  | 1.0        | 1.0           | 115019.61          | 20               |
| catalog | 1001001.0  | 1.0        | 2.0           | 146344.47          | 27               |
| catalog | 1001001.0  | 1.0        | 3.0           | 22597.190000000002 | 3                |
| catalog | 1001001.0  | 1.0        | 4.0           | 107555.43000000001 | 23               |
| catalog | 1001001.0  | 1.0        | 5.0           | 122521.31000000001 | 25               |
| catalog | 1001001.0  | 1.0        | 6.0           | 16883.97           | 3                |
| catalog | 1001001.0  | 1.0        | 7.0           | 46329.780000000006 | 9                |
| catalog | 1001001.0  | 1.0        | 8.0           | 77861.85           | 13               |
| catalog | 1001001.0  | 1.0        | 9.0           | 99985.34999999999  | 21               |
| catalog | 1001001.0  | 1.0        | 10.0          | 100105.28          | 23               |
| catalog | 1001001.0  | 2.0        |               | 68565.38           | 14               |
| catalog | 1001001.0  | 2.0        |               | 43967.969999999994 | 7                |
| catalog | 1001001.0  | 2.0        |               | 12633.87           | 3                |
| catalog | 1001001.0  | 2.0        | 2.0           | 43967.969999999994 | 7                |
| catalog | 1001001.0  | 2.0        | 3.0           | 68565.38           | 14               |
| catalog | 1001001.0  | 2.0        | 5.0           | 12633.87           | 3                |
| catalog | 1001001.0  | 3.0        |               | 60551.64000000001  | 14               |
| catalog | 1001001.0  | 3.0        |               | 14426.92           | 4                |
| catalog | 1001001.0  | 3.0        |               | 36821.61           | 7                |
| catalog | 1001001.0  | 3.0        |               | 30078.07           | 3                |
| catalog | 1001001.0  | 3.0        |               | 28455.23           | 4                |
| catalog | 1001001.0  | 3.0        |               | 28351.61           | 11               |
| catalog | 1001001.0  | 3.0        | 1.0           | 11100.79           | 5                |
| catalog | 1001001.0  | 3.0        | 2.0           | 60551.64000000001  | 14               |
| catalog | 1001001.0  | 3.0        | 4.0           | 28455.23           | 4                |
| catalog | 1001001.0  | 3.0        | 6.0           | 36821.61           | 7                |
| catalog | 1001001.0  | 3.0        | 7.0           | 17250.82           | 6                |
| catalog | 1001001.0  | 3.0        | 8.0           | 14426.92           | 4                |
| catalog | 1001001.0  | 3.0        | 9.0           | 30078.07           | 3                |
| catalog | 1001001.0  | 4.0        |               | 47553.200000000004 | 10               |
| catalog | 1001001.0  | 4.0        |               | 45473.85           | 13               |
| catalog | 1001001.0  | 4.0        |               | 16558.92           | 8                |
| catalog | 1001001.0  | 4.0        | 2.0           | 45473.85           | 13               |
| catalog | 1001001.0  | 4.0        | 3.0           | 16558.92           | 8                |
| catalog | 1001001.0  | 4.0        | 4.0           | 47553.200000000004 | 10               |
| catalog | 1001001.0  | 5.0        |               | 29678.5            | 5                |
| catalog | 1001001.0  | 5.0        |               | 30112.109999999997 | 12               |
| catalog | 1001001.0  | 5.0        | 9.0           | 30112.109999999997 | 12               |
| catalog | 1001001.0  | 5.0        | 10.0          | 29678.5            | 5                |
| catalog | 1001001.0  | 6.0        |               | 10261.82           | 3                |
| catalog | 1001001.0  | 6.0        | 9.0           | 10261.82           | 3                |
| catalog | 1001001.0  | 7.0        |               | 18244.94           | 3                |
| catalog | 1001001.0  | 7.0        | 7.0           | 18244.94           | 3                |
| catalog | 1001001.0  | 8.0        |               | 26895.97           | 6                |
+---------+------------+------------+---------------+--------------------+------------------+

Not 100% sure yet what's the issue exactly... I'm still triple checking that it's not a problem with datafusion-distributed testing setup

[alamb]

Here is a backport for the LimitedBatchCoalescer fix:

• [branch-52] Fix Internal error: Assertion failed: !self.finished: LimitedBatchCoalescer (#19785) #19836

Given @gabotechs is looking into something else, I plan to wait another few days before making a release candidate to see if we need to backport anything else.

[gabotechs]

I've created a proper bug report with steps to reproduce here:

• Different results in TPC-DS q14 depending on the number of partitions #19849

I can confirm that this is a regression introduced in version 52

[alamb]

I've created a proper bug report with steps to reproduce here:

• Different results in TPC-DS q14 depending on the number of partitions #19849

I can confirm that this is a regression introduced in version 52

Thank you -- added to the list at the top of this ticket