Skip to content

osdc/Objecter: do not keep a stale crush_location after a change - #73729

Draft
aainscow wants to merge 1 commit into
ceph:mainfrom
aainscow:wip-objecter-crush-location-stale
Draft

aainscow wants to merge 1 commit into
ceph:mainfrom
aainscow:wip-objecter-crush-location-stale

Conversation

@aainscow

Copy link
Copy Markdown
Contributor

The Objecter and the CephContext both observe crush_location. On a change, the Objecter copies the CephContext's parsed location, but the config proxy calls a change's observers in the order of their addresses, not the order they were added in. When the Objecter's observer runs first, it copies the previous location, so a client whose crush_location changes at run time stays one change behind and sends localized reads to the OSDs of its old location.

The Objecter now parses the option in its own observer before copying it. The new unit test runs the Objecter's observer before the CephContext's; it fails without the fix and passes with it. unittest_splitop now links unit-main, so the test has the global CephContext it needs.

Found by the librados zone split-read tests of #72643, which move their client between zones and failed in about 1 of 5 runs.

Fixes: https://tracker.ceph.com/issues/82994

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

🤖 Raised by Claude Code.

The Objecter and the CephContext both observe crush_location. The
Objecter copies the CephContext's parsed CrushLocation, but the config
proxy calls the observers of a change in the order of their addresses,
not the order in which they were added. When the Objecter's observer
runs first, it copies the previous location, so a client whose
crush_location changes at run time can keep sending localized reads to
the OSDs of its old location.

Parse the option in the Objecter's observer before copying it.

unittest_splitop links unit-main, so that the new test has the global
CephContext it needs.

Found by the librados zone split-read tests, which move their client
between zones and failed intermittently.

Fixes: https://tracker.ceph.com/issues/82994
Assisted-by: Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant