Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: databricks/databricks-sql-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: databricks/databricks-sql-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: ci/optimize-e2e-and-coverage-v2
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 3 commits
  • 8 files changed
  • 1 contributor

Commits on Apr 13, 2026

  1. Optimize CI: consolidate workflows, fix caching, speed up e2e tests

    Workflow consolidation:
    - Delete integration.yml and daily-telemetry-e2e.yml (redundant with
      coverage workflow which already runs all e2e tests)
    - Add push-to-main trigger to coverage workflow
    - Run all tests (including telemetry) in single pytest invocation with
      --dist=loadgroup to respect xdist_group markers for isolation
    
    Fix pyarrow cache:
    - Remove cache-path: .venv-pyarrow from pyarrow jobs. Poetry always
      creates .venv regardless of the cache-path input, so the cache was
      never saved ("Path does not exist" error). The cache-suffix already
      differentiates keys between variants.
    
    Fix 3.14 post-test DNS hang:
    - Add enable_telemetry=False to unit test DUMMY_CONNECTION_ARGS that
      use server_hostname="foo". This prevents FeatureFlagsContext from
      making real HTTP calls to fake hosts, eliminating ~8min hang from
      ThreadPoolExecutor threads timing out on DNS on protected runners.
    
    Improve e2e test parallelization:
    - Split TestPySQLLargeQueriesSuite into 3 separate classes
      (TestPySQLLargeWideResultSet, TestPySQLLargeNarrowResultSet,
      TestPySQLLongRunningQuery) so xdist distributes them across workers
      instead of all landing on one.
    
    Speed up slow tests:
    - Reduce large result set sizes from 300MB to 100MB (still validates
      large fetches, lz4, chunking, row integrity)
    - Start test_long_running_query at scale_factor=50 instead of 1 to
      skip ramp-up iterations that finish instantly
    
    Co-authored-by: Isaac
    Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
    vikrantpuppala committed Apr 13, 2026
    Configuration menu
    Copy the full SHA
    1625263 View commit details
    Browse the repository at this point in the history
  2. Further optimize e2e: 4 workers, lower long-query threshold, split lz4

    - Use -n 4 instead of -n auto in coverage workflow. The e2e tests are
      network-bound (waiting on warehouse), not CPU-bound, so 4 workers on
      a 2-CPU runner is fine and doubles parallelism.
    - Lower test_long_running_query min_duration from 3 min to 1 min.
      The test validates long-running query completion — 1 minute is
      sufficient and saves ~4 min per variant.
    - Split lz4 on/off loop in test_query_with_large_wide_result_set into
      separate parametrized test cases so xdist can run them on different
      workers instead of sequentially in one test.
    
    Co-authored-by: Isaac
    Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
    vikrantpuppala committed Apr 13, 2026
    Configuration menu
    Copy the full SHA
    ba8be42 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2026

  1. Address review: inline test methods, drop mixin pattern

    Per review feedback from jprakash-db:
    - Remove mixin classes (LargeWideResultSetMixin, etc) — inline the
      test methods directly into the test classes in test_driver.py
    - Remove backward-compat LargeQueriesMixin alias (nothing uses it)
    - Rename _LargeQueryRowHelper — replaced entirely by inlining
    - Convert large_queries_mixin.py to just a fetch_rows() helper function
    
    Co-authored-by: Isaac
    Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
    vikrantpuppala committed Apr 14, 2026
    Configuration menu
    Copy the full SHA
    21d475b View commit details
    Browse the repository at this point in the history
Loading