Repository navigation
client: accept Cap re-issue and drop stale CapReleases - #73732
edwinzrodriguez wants to merge 1 commit into
Conversation
Under heavy trim, the client can queue a CapRelease, re-acquire the same inode Cap (e.g. via lookup) before the RELEASE is flushed, then keep the Cap locally while the MDS still honors the late RELEASE and issues a new cap_id at seq 1. A later readdir Cap update hits add_update_cap's stale path (seq <= local seq) with a mismatched cap_id and aborts on ceph_assert(cap.cap_id == cap_id). Only treat seq <= cap.seq as a delayed update when cap_id still matches; otherwise accept the MDS Cap re-issue. On Cap (re)acquire, cancel any unflushed CapRelease items for that ino and complete the deferred pinned-icaps accounting so a RELEASE cannot race past re-issue. Fixes: https://tracker.ceph.com/issues/81523 Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>
bf69c75 to
621808b
Compare
gregsfortytwo
left a comment
There was a problem hiding this comment.
You’re saying the client can tell the MDS it’s dropping a cap (eg Fx) while actually holding a reference locally and relying on it?
Or is this all entirely a local memory thing? I’m not sure how the MDS could be honoring a release if it never sees it, which suggests to me that this is something we need to fix on the front end, not patch up when we discover the conflict. Because if we tell the MDS we’ve dropped a cap while continuing to use it, all our consistency guarantees go out the window…
Or am I misunderstanding?
|
No, this is a result of a race in the mds. Details are in https://tracker.ceph.com/issues/81523. I hit this reproducing a SpecStorage 2020-style metadata storm through libcephfs/ceph-fuse with aggressive client cache trim. Under the trim the client queues many CapReleases. A race lets the client re-acquire the same Cap (via lookup) before the RELEASE is flushed; the MDS later honors the late RELEASE and drops the Cap, then later issues a new cap_id at seq=1 (e.g. on readdir). The client still has the old Cap with higher seq, so add_update_cap takes the stale-update path and asserts on cap_id mismatch. (The kernel client WARNs on the analogous path and continues the stale merge, which leaves Cap identity desynced.) The pending-RELEASE cancel on re-acquire is meant to stop the RELEASE from being sent after we hold the Cap again. |
|
This problem can be reproduced either with SpecStorage 2020 with the kernel client or using cephfs-mdsbench from #73737 |
Under heavy trim, the client can queue a CapRelease, re-acquire the same inode Cap (e.g. via lookup) before the RELEASE is flushed, then keep the Cap locally while the MDS still honors the late RELEASE and issues a new cap_id at seq 1. A later readdir Cap update hits add_update_cap's stale path (seq <= local seq) with a mismatched cap_id and aborts on ceph_assert(cap.cap_id == cap_id).
Only treat seq <= cap.seq as a delayed update when cap_id still matches; otherwise accept the MDS Cap re-issue. On Cap (re)acquire, cancel any unflushed CapRelease items for that ino and complete the deferred pinned-icaps accounting so a RELEASE cannot race past re-issue.
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an
xbetween the brackets:[x]. Spaces and capitalization matter when checking off items this way.Checklist
Show available Jenkins commands
jenkins test classic perfJenkins Job | Jenkins Job Definitionjenkins test crimson perfJenkins Job | Jenkins Job Definitionjenkins test signedJenkins Job | Jenkins Job Definitionjenkins test make checkJenkins Job | Jenkins Job Definitionjenkins test make check arm64Jenkins Job | Jenkins Job Definitionjenkins test submodulesJenkins Job | Jenkins Job Definitionjenkins test dashboardJenkins Job | Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job | Jenkins Job Definitionjenkins test apiJenkins Job | Jenkins Job Definitionjenkins test docsReadTheDocs | Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs | Jenkins Jobs Definitionjenkins test windowsJenkins Job | Jenkins Job Definitionjenkins test rook e2eJenkins Job | Jenkins Job DefinitionYou must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.