Skip to content

test/librados: keep the cluster usable when creating an EC pool fails - #73740

Draft
aainscow wants to merge 1 commit into
ceph:mainfrom
aainscow:wip-librados-ec-cleanup-after-shutdown
Draft

aainscow wants to merge 1 commit into
ceph:mainfrom
aainscow:wip-librados-ec-cleanup-after-shutdown

Conversation

@aainscow

Copy link
Copy Markdown
Contributor

create_ec_pool() and create_ec_pool_pp() work on a cluster handle that the caller owns and keeps using, but most of their error paths shut it down. After a failed osd pool set <pool> allow_ec_optimizations true, the error path also sends another mon command on the shut-down cluster. In C this uses the freed RadosClient and is followed by a second rados_shutdown(). That path's pool delete also looks the new pool up in a client osdmap that does not have it yet, so the pool is left behind. Whenever the set fails, the EC test binaries crash instead of reporting a failure.

The helpers leave the caller's cluster connected on every error path. create_one_ec_pool() shuts down the cluster it connected, as create_one_ec_pool_pp() does. The allow_ec_optimizations error path waits for the latest osdmap and removes the pool, profile and rule with destroy_ec_pool() or destroy_ec_pool_pp().

Checked on one mon whose default EC profile cannot take EC optimizations, so the set always fails. Without the fix ceph_test_rados_api_io_pp segfaults, and valgrind shows ceph_test_rados_api_aio and ceph_test_rados_api_watch_notify using the freed RadosClient. With it all three fail cleanly with the monitor's error, valgrind reports 0 errors, and no test pools are left.

Fixes: https://tracker.ceph.com/issues/83011

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

🤖 Raised by Claude Code.

create_ec_pool() and create_ec_pool_pp() work on a cluster handle that
the caller owns and keeps using, but most of their error paths shut it
down. The error path for a failed "osd pool set allow_ec_optimizations"
is worse: it shuts the cluster down and then sends another mon command
on it (in C on the freed handle, followed by a second rados_shutdown()),
and its pool delete looks the new pool up in a client osdmap that does
not contain it yet, so the pool is never removed. Whenever that set
fails, for example when a resent "osd pool create" is answered before
the pool is committed, the EC test binaries crash instead of reporting
a test failure.

Leave the cluster connected on every error path of create_ec_pool() and
create_ec_pool_pp(). create_one_ec_pool() shuts down the cluster it
connected when the pool cannot be created, as create_one_ec_pool_pp()
does. On the allow_ec_optimizations error path, wait for the latest
osdmap and remove the pool, profile and rule with destroy_ec_pool() or
destroy_ec_pool_pp().

Fixes: https://tracker.ceph.com/issues/83011
Assisted-by: Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant