feature: add `PulpProject` class for Pulp-based package repository support by vamahaja · Pull Request #2163 · ceph/teuthology

vamahaja · 2026-03-26T13:33:55Z

This PR adds a PulpProject class that extends GitbuilderProject so teuthology can use Pulp-hosted package repositories. It introduces new config use_artifacts, set it to pulp (globally in site config or per job) to use Pulp; other values keep the current Gitbuilder and Shaman behavior.

Configuration

use_artifacts: 'pulp' - selects the Pulp backend (alongside existing 'shaman' and the default Gitbuilder path when unset / other values).
pulp_host — Pulp server hostname (documented in siteconfig.rst).

Key changes

Implement PulpProject inheriting from GitbuilderProject.
Override _get_base_url() to use Pulp’s repository API.
Override _get_distro() for Pulp’s distro format.
Override _get_package_version() and _get_package_sha1() to read metadata from Pulp responses.
Update get_builder_project() so config.use_artifacts == 'pulp' returns PulpProject.
Update siteconfig (and related docs) for use_artifacts: 'pulp' and pulp_host.
Add mock test TestPulpProject to teuthology/test/test_packaging.py.

kshtsk · 2026-03-27T19:50:34Z

This looks good, do we have an instance running with pulp to check this PR?

vamahaja · 2026-03-30T04:30:39Z

This looks good, do we have an instance running with pulp to check this PR?

@kshtsk for review, I am working on deploying Pulp server on OCP cluster.

batrick

Looking good!!

batrick

I recommend using an LLM to help you write up some docs on this new change.

vamahaja · 2026-05-25T06:31:21Z

I recommend using an LLM to help you write up some docs on this new change.

The draft documentation is currently available on CephPad: https://pad.ceph.com/p/ceph_artifact_management_via_pulp.

I will migrate this to a wiki page once all associated tasks are complete.

vamahaja · 2026-06-05T13:50:01Z

Smoke tests are passed - http://172.21.5.153:9091/vamahaja-2026-06-05_08:39:01-smoke-tentacle-distro-default-trial

djgalloway · 2026-06-05T14:39:52Z

This seems like a breaking change... While the number of global teuthology users is low, I am personally not a fan of removing options like use_shaman that have been available for years.

I'd prefer to see something like

use_shaman: false
use_pulp: true

But this is just my opinion. I'd like @zmc to weigh in.

Can the option be set on a per-job/run basis?

kshtsk · 2026-06-05T14:48:55Z

This seems like a breaking change... While the number of global teuthology users is low, I am personally not a fan of removing options like use_shaman that have been available for years.

I don't think it is that breaking, @djgalloway to avoid interruption, could you or someone add the config to the server's teuthology yaml already:

package_source: shaman
shaman:
      endpoint: "https://shaman.ceph.com/api/"
      force_noarch: True

Or at least we need this before the merge.

I'd prefer to see something like
use_shaman: false
use_pulp: true

This complicates the code logic and will make potential misuse, what to prefer when both "use_shaman" and "use_pulp" are enable or disabled, it is not possible to predict which is used not looking into the source code.

But this is just my opinion. I'd like @zmc to weigh in.

Can the option be set on a per-job/run basis?

@djgalloway I like the idea of giving package_source on run/job basis though.

vamahaja · 2026-06-05T16:33:41Z

This seems like a breaking change... While the number of global teuthology users is low, I am personally not a fan of removing options like use_shaman that have been available for years.

I don't think it is that breaking, @djgalloway to avoid interruption, could you or someone add the config to the server's teuthology yaml already:
package_source: shaman
shaman:
      endpoint: "https://shaman.ceph.com/api/"
      force_noarch: True
Or at least we need this before the merge.
I'd prefer to see something like
use_shaman: false
use_pulp: true
This complicates the code logic and will make potential misuse, what to prefer when both "use_shaman" and "use_pulp" are enable or disabled, it is not possible to predict which is used not looking into the source code.

But this is just my opinion. I'd like @zmc to weigh in.
Can the option be set on a per-job/run basis?

@djgalloway I like the idea of giving package_source on run/job basis though.

@djgalloway @kshtsk no changes are needed, default values are already pointing to shaman with the required configs - https://github.com/ceph/teuthology/pull/2163/changes#diff-4525b7398914ab0a466329cd35d0e8b22b35479597fbfdca524baa871ae7d184R184

zmc

I think this looks great - I see only one outstanding item from @batrick, but other than that the PR seems ready to me!

…pport Add a `PulpProject` subclass of `GitbuilderProject` so teuthology can use Pulp-hosted package repos when `config.package_source` is `pulp`, alongside the existing `Gitbuilder` and `Shaman` artifact backends. Signed-off-by: Vaibhav Mahajan <vamahaja@redhat.com>

zmc

Looks like all review items have been addressed, so from my perspective it looks ready - @batrick would you take a look as well, please?

batrick · 2026-06-10T01:23:00Z

Yes, in the current setup, the ceph-ci-bot user is created with read-only permissions for API access.

For this to work long-term I don't know how we cannot have pulp be publicly (on the VPN at least) readable?

batrick

I'm okay with this as-is to start.

kshtsk · 2026-06-10T15:28:07Z

How's pushing the big green button?

vamahaja requested a review from a team as a code owner March 26, 2026 13:33

vamahaja requested review from VallariAg and kamoltat and removed request for a team March 26, 2026 13:33

vamahaja added DNM feature labels Mar 26, 2026

vamahaja marked this pull request as draft March 26, 2026 13:34

vamahaja mentioned this pull request Mar 26, 2026

Alternate package providers #2135

Closed

vamahaja linked an issue Mar 26, 2026 that may be closed by this pull request

Alternate package providers #2135

Closed

deepssin requested review from amathuria, kshtsk and zmc March 27, 2026 04:15

vamahaja force-pushed the pulp-project branch 5 times, most recently from f60b0ef to b2e5898 Compare April 7, 2026 09:19

kshtsk reviewed Apr 7, 2026

View reviewed changes

Comment thread teuthology/openstack/setup-openstack.sh Outdated

kshtsk reviewed Apr 7, 2026

View reviewed changes

Comment thread teuthology/config.py Outdated

kshtsk reviewed Apr 7, 2026

View reviewed changes

Comment thread teuthology/config.py Outdated

vamahaja force-pushed the pulp-project branch 2 times, most recently from 6a38b81 to bb9ccff Compare April 7, 2026 11:58

batrick requested changes Apr 9, 2026

View reviewed changes

Comment thread teuthology/packaging.py Outdated

Comment thread teuthology/run.py Outdated

vamahaja force-pushed the pulp-project branch 4 times, most recently from 0e9add6 to e27a58d Compare May 4, 2026 09:22

vamahaja force-pushed the pulp-project branch from e27a58d to 1f982d4 Compare May 12, 2026 14:02

vamahaja force-pushed the pulp-project branch 8 times, most recently from 39c888c to c68e408 Compare May 21, 2026 11:34

vamahaja removed the DNM label May 21, 2026

vamahaja marked this pull request as ready for review May 21, 2026 12:58

vamahaja requested a review from batrick May 21, 2026 12:58

batrick requested changes May 21, 2026

View reviewed changes

Comment thread teuthology/packaging.py Outdated

Comment thread teuthology/task/kernel.py Outdated

Comment thread docs/detailed_test_config.rst Outdated

Comment thread docs/detailed_test_config.rst

Comment thread docs/detailed_test_config.rst

vamahaja force-pushed the pulp-project branch from c68e408 to 4f1a272 Compare May 25, 2026 06:28

vamahaja force-pushed the pulp-project branch 3 times, most recently from 5e367d6 to b133a48 Compare June 1, 2026 09:31

zmc reviewed Jun 5, 2026

View reviewed changes

vamahaja force-pushed the pulp-project branch from b133a48 to 0e71339 Compare June 8, 2026 06:09

vamahaja requested a review from zmc June 8, 2026 08:53

zmc approved these changes Jun 10, 2026

View reviewed changes

batrick approved these changes Jun 10, 2026

View reviewed changes

zmc merged commit aaa2c68 into ceph:main Jun 10, 2026
10 checks passed

Uh oh!

Conversation

vamahaja commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kshtsk commented Mar 27, 2026

Uh oh!

vamahaja commented Mar 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vamahaja commented May 25, 2026

Uh oh!

vamahaja commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

djgalloway commented Jun 5, 2026

Uh oh!

kshtsk commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vamahaja commented Jun 5, 2026

Uh oh!

zmc left a comment

Choose a reason for hiding this comment

Uh oh!

zmc left a comment

Choose a reason for hiding this comment

Uh oh!

batrick commented Jun 10, 2026

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

kshtsk commented Jun 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vamahaja commented Mar 26, 2026 •

edited

Loading

vamahaja commented Jun 5, 2026 •

edited

Loading

kshtsk commented Jun 5, 2026 •

edited

Loading