Skip to content

Tsaucer/prepare tpch examples for ci#710

Closed
timsaucer wants to merge 49 commits into
apache:mainfrom
timsaucer:tsaucer/prepare_tpch_examples_for_ci
Closed

Tsaucer/prepare tpch examples for ci#710
timsaucer wants to merge 49 commits into
apache:mainfrom
timsaucer:tsaucer/prepare_tpch_examples_for_ci

Conversation

@timsaucer
Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #696.

Rationale for this change

This PR sets up a work flow to generate TPH-C 1Gb data set in CI, runs the 22 examples, and compares their results to the
known answer file. By adding this PR we improve the robustness of our test suite.

What changes are included in this PR?

This PR adds the following changes:

  • Adds CI workflow to generate TPH-C data set and answer file
  • Adds pytest file to run all 22 examples and compare them to the answer file
  • Exposes substring operation that was missing. Sometime between 36.0.0 and 38.0.0 substr was performing the operation instead.

Are there any user-facing changes?

substring function is exposed in python.


- name: Run TPC-H examples
run: |
cd examples/tpch
Copy link
Copy Markdown
Contributor

@Michael-J-Ward Michael-J-Ward May 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the first line here should activate the venv

      - name: Run TPC-H examples
        run: |
          source venv/bin/activate
          cd examples/tpch
          python convert_data_to_parquet.py
          pytest _tests.py

Copy link
Copy Markdown
Member Author

@timsaucer timsaucer May 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I hadn't looked at how far it got before I had to step away for the day. I know I also need to add in the upload and download artifacts between stages.

@timsaucer
Copy link
Copy Markdown
Member Author

It looks like CI is running correctly and also caching the data. I’ll rebase in the morning and get the PR ready to merge.

@timsaucer
Copy link
Copy Markdown
Member Author

Closing in favor of #711

@timsaucer timsaucer closed this May 21, 2024
@timsaucer timsaucer deleted the tsaucer/prepare_tpch_examples_for_ci branch May 23, 2024 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure examples stay updated in CI.

2 participants