Commits

Commit history

1. Commits on Oct 18, 2026

   • [SPARK-60294][SQL] Disable Arrow-backed columnar input to Python UDFs by default

     Show description for 2487c98
     HyukjinKwon

     committed
     2487c98View commit details

     Copy full SHA for 2487c98

     View code at this pointBrowse repository at this point
2. Commits on Oct 16, 2026

   • [SPARK-60245][SQL] Support ARRAY<DOUBLE> input in vector_sum

     Show description for dcfe227
     

     urosstan-db

     authored and

     uros-b

     committed
     dcfe227View commit details

     Copy full SHA for dcfe227

     View code at this pointBrowse repository at this point

   • [SPARK-60275][PYTHON] Migrate SQL_BATCHED_UDF to the EvalTypeHandler pipeline and remove read_udfs

     Show description for 4371ca7
     Yicong-Huang

     committed
     4371ca7View commit details

     Copy full SHA for 4371ca7

     View code at this pointBrowse repository at this point

   • [SPARK-60238][PYTHON] Migrate stateful streaming UDF eval types to the EvalTypeHandler pipeline

     Show description for f39493b
     Yicong-Huang

     committed
     f39493bView commit details

     Copy full SHA for f39493b

     View code at this pointBrowse repository at this point
3. Commits on Oct 15, 2026

   • [MINOR][ML] Correct KMeans.setIntermediateStorageLevel version metadata

     Show description for d213c88
     zhengruifeng

     committed
     d213c88View commit details

     Copy full SHA for d213c88

     View code at this pointBrowse repository at this point
4. Commits on Oct 14, 2026

   • [SPARK-60235][PYTHON] Migrate element-wise UDF eval types to the EvalTypeHandler pipeline

     Show description for ff0ddf8
     Yicong-Huang

     committed
     ff0ddf8View commit details

     Copy full SHA for ff0ddf8

     View code at this pointBrowse repository at this point

   • [SPARK-60219][PYTHON] Migrate SQL_ARROW_BATCHED_UDF to the EvalTypeHandler pipeline

     Show description for 19ce8f5
     Yicong-Huang

     committed
     19ce8f5View commit details

     Copy full SHA for 19ce8f5

     View code at this pointBrowse repository at this point

   • [SPARK-60218][PYTHON] Migrate SQL_TABLE_UDF to the EvalTypeHandler pipeline and remove read_udtf

     Show description for 18b5254
     Yicong-Huang

     committed
     18b5254View commit details

     Copy full SHA for 18b5254

     View code at this pointBrowse repository at this point

   • [SPARK-60220][PYTHON] Add ASV benchmarks for SQL_TABLE_UDF eval type

     Show description for bcc9019
     Yicong-Huang

     committed
     bcc9019View commit details

     Copy full SHA for bcc9019

     View code at this pointBrowse repository at this point
5. Commits on Oct 13, 2026

   • [SPARK-60209][PYTHON] Migrate SQL_ARROW_TABLE_UDF to the EvalTypeHandler pipeline

     Show description for 7d92cce
     Yicong-Huang

     committed
     7d92cceView commit details

     Copy full SHA for 7d92cce

     View code at this pointBrowse repository at this point

   • [SPARK-60208][PYTHON] Migrate grouped aggregate and window pandas UDF eval types to pandas eval type handlers

     Show description for 8ac4e17
     Yicong-Huang

     committed
     8ac4e17View commit details

     Copy full SHA for 8ac4e17

     View code at this pointBrowse repository at this point

   • [SPARK-59909][PYTHON] Fix PySpark errors raised with undefined error conditions or mismatched message parameters

     Show description for 4c53b2a
     

     jubins

     authored and

     cloud-fan

     committed
     4c53b2aView commit details

     Copy full SHA for 4c53b2a

     View code at this pointBrowse repository at this point

   • [SPARK-60207][PYTHON][TEST] Move eval type handler test helpers onto the test classes

     Show description for e240540
     Yicong-Huang

     committed
     e240540View commit details

     Copy full SHA for e240540

     View code at this pointBrowse repository at this point

   • [MINOR][PYTHON][TEST] Make column resolution error differences explicit

     Show description for f70e229
     

     marko-radoicic-db

     authored and

     zhengruifeng

     committed
     f70e229View commit details

     Copy full SHA for f70e229

     View code at this pointBrowse repository at this point
6. Commits on Oct 12, 2026

   • [SPARK-60069][PYTHON] Migrate SQL_ARROW_UDTF to the EvalTypeHandler pipeline

     Show description for 770c6da
     Yicong-Huang

     committed
     770c6daView commit details

     Copy full SHA for 770c6da

     View code at this pointBrowse repository at this point

   • [SPARK-60100][PYTHON] Migrate grouped and cogrouped map pandas UDF eval types to pandas eval type handlers

     Show description for 34a5e22
     Yicong-Huang

     committed
     34a5e22View commit details

     Copy full SHA for 34a5e22

     View code at this pointBrowse repository at this point
7. Commits on Oct 11, 2026

   • [SPARK-60122][PYTHON] Remove non-existent `avro` Maven profile from Avro doctest skip messages

     Show description for 4def966
     dongjoon-hyun

     committed
     4def966View commit details

     Copy full SHA for 4def966

     View code at this pointBrowse repository at this point

   • [SPARK-60159][PYTHON][TEST] Add tests for string and DataFrame keyword arguments in spark.sql formatting

     Show description for e81f9af
     

     LearnFirst

     authored and

     gaogaotiantian

     committed
     e81f9afView commit details

     Copy full SHA for e81f9af

     View code at this pointBrowse repository at this point

   • [SPARK-60151][ML][PYTHON] Fix CrossValidatorModel.load returning the last fold's sub-models for every fold

     Show description for 1a32dca
     

     chinaoel

     authored and

     gaogaotiantian

     committed
     1a32dcaView commit details

     Copy full SHA for 1a32dca

     View code at this pointBrowse repository at this point

   • [SPARK-60154][PYTHON][TEST] Add test for Catalog.createDatabase with properties

     Show description for c2130c9
     

     kjs1095

     authored and

     gaogaotiantian

     committed
     c2130c9View commit details

     Copy full SHA for c2130c9

     View code at this pointBrowse repository at this point

   • [SPARK-60141][PYTHON] Correct SparkContext profiler collector type annotation

     Show description for 2c17e77
     

     Horse-MA00

     authored and

     gaogaotiantian

     committed
     2c17e77View commit details

     Copy full SHA for 2c17e77

     View code at this pointBrowse repository at this point

   • [SPARK-60145][PYTHON] Remove type: ignore[var-annotated] in pyspark.sql

     Show description for 37913ef
     

     soaring0616

     authored and

     gaogaotiantian

     committed
     37913efView commit details

     Copy full SHA for 37913ef

     View code at this pointBrowse repository at this point

   • [SPARK-59985][ML][PYTHON] Fix malformed warning in TorchDistributor w…

     Show description for d7635d0
     

     firasbouzazi

     authored and

     viirya

     committed
     d7635d0View commit details

     Copy full SHA for d7635d0

     View code at this pointBrowse repository at this point
8. Commits on Oct 8, 2026

   • [SPARK-59275][PYTHON] Reject CHAR/VARCHAR at Python/Arrow host boundaries

     Show description for fd4d816
     

     srielau

     authored and

     dtenedor

     committed
     fd4d816View commit details

     Copy full SHA for fd4d816

     View code at this pointBrowse repository at this point
9. Commits on Oct 7, 2026

   • [SPARK-59959][PYTHON] Migrate SQL_SCALAR_PANDAS_ITER_UDF and SQL_MAP_PANDAS_ITER_UDF to pandas eval type handlers

     Show description for 37d0691
     Yicong-Huang

     committed
     37d0691View commit details

     Copy full SHA for 37d0691

     View code at this pointBrowse repository at this point
10. Commits on Oct 6, 2026

    • [MINOR][PYTHON] Coerce the JVM error message to str in ArrowCollectSerializer.load_stream

      Show description for 29b60f5
      

      Spenserrrr

      authored and

      Yicong-Huang

      committed
      29b60f5View commit details

      Copy full SHA for 29b60f5

      View code at this pointBrowse repository at this point
11. Commits on Oct 2, 2026

    • [SPARK-59911][PYTHON] Migrate SQL_SCALAR_PANDAS_UDF to a pandas eval type handler

      Show description for 49a8389
      Yicong-Huang

      committed
      49a8389View commit details

      Copy full SHA for 49a8389

      View code at this pointBrowse repository at this point

    • [SPARK-59942][PYTHON] Use real RunnerConf in Arrow eval type handler tests

      Show description for 523ab88
      Yicong-Huang

      committed
      523ab88View commit details

      Copy full SHA for 523ab88

      View code at this pointBrowse repository at this point

    • [SPARK-44462][SS][CONNECT] Fix the session passed to foreachBatch in Spark Connect

      Show description for 362ee55
      

      2 peopleauthored and

      HeartSaVioR

      committed
      362ee55View commit details

      Copy full SHA for 362ee55

      View code at this pointBrowse repository at this point
12. Commits on Oct 1, 2026

    • [SPARK-59912][PYTHON][TEST] Fix flaky WorkerReuseTest and WorkerPoolCrashTest by pinning spark.python.use.daemon

      Show description for 0846225
      gaogaotiantian

      committed
      0846225View commit details

      Copy full SHA for 0846225

      View code at this pointBrowse repository at this point

    • [SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the mapInBatch acceptAnyIterable default in mapInArrow/mapInPandas

      Show description for a363a85
      Yicong-Huang

      committed
      a363a85View commit details

      Copy full SHA for a363a85

      View code at this pointBrowse repository at this point

    • [SPARK-59727][PYTHON] Migrate Arrow incremental aggregate and window UDF eval types to the EvalTypeHandler pipeline

      Show description for 2b1d78b
      Yicong-Huang

      committed
      2b1d78bView commit details

      Copy full SHA for 2b1d78b

      View code at this pointBrowse repository at this point
13. Commits on Sep 30, 2026

    • [SPARK-59773][PYTHON] Report Python worker metrics as JSON

      Show description for e85102e
      

      Spenserrrr

      authored and

      gaogaotiantian

      committed
      e85102eView commit details

      Copy full SHA for e85102e

      View code at this pointBrowse repository at this point

    • [SPARK-59885][PYTHON][TEST] Fix flaky WorkerMemoryTest by pinning spark.task.cpus

      Show description for f27eab5
      gaogaotiantian

      committed
      f27eab5View commit details

      Copy full SHA for f27eab5

      View code at this pointBrowse repository at this point
14. Commits on Sep 29, 2026

    • [SPARK-59847][CONNECT][SS][PYTHON] Fix deadlock in StreamingQueryListenerBus when removing the last listener

      Show description for b2dd3e4
      gaogaotiantian

      committed
      b2dd3e4View commit details

      Copy full SHA for b2dd3e4

      View code at this pointBrowse repository at this point