Repository navigation
Conversation
…dirfrag With mds_dir_prefetch, the default, a lookup that misses a dentry loads the whole dirfrag. That pays off when the cache can keep the dirfrag until its other entries are looked up. Once a directory tree is larger than the cache, it does not. The cache sits at its trim target, so each loaded dirfrag pushes other entries out, and its own entries are trimmed again before most of them are used. Every lookup that misses then reads a whole dirfrag from the metadata pool and puts thousands of inodes through the cache. In mdtest stat phases over 12M files with 2 active MDS and a 16 GiB cache, a rank fetched 275 whole dirfrags and trimmed 626k inodes per 10 s while replying to only 3k lookups. Load only the wanted dentry when the dirfrag would not fit: - the cache is over its trim target, or - it has lately trimmed at least as many entries as the dirfrag holds. Trimming then keeps the cache just below its target, and a loaded dirfrag only pushes other entries out again, as in mdtest delete phases, or - the dirfrag would take the cache over its target. An entry is taken to be as large as a cached entry on average. While the cache holds fewer entries than the dirfrag, that average says little, and the whole dirfrag is loaded only while the cache is less than half full. The number of entries comes from the dirfrag's fragstat, or from the directory's dirstat when a dirfrag closed by trimming comes back without its fnode. Fixes: https://tracker.ceph.com/issues/82988 Reported-by: Anthony D'Atri <anthony.datri@gmail.com> Assisted-by: LLM Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
With room in the cache, a lookup that misses loads the whole dirfrag. With the cache over its trim target, it loads only the wanted dentry, and readdir still loads the whole dirfrag. Fixes: https://tracker.ceph.com/issues/82988 Assisted-by: LLM Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
Config Diff Tool Output! changed: mds_dir_prefetch: old: (mds.yaml.in)
! changed: mds_dir_prefetch: new: When a lookup misses a dentry, load the whole dirfrag instead of only that dentry. When the MDS cache has no room for the whole dirfrag, only the dentry is loaded. (mds.yaml.in)
The above configuration changes are found in the PR. Please update the relevant release documentation if necessary. |
4 of 14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With mds_dir_prefetch (the default), a lookup that misses a dentry makes CDir::fetch() load the whole dirfrag, not only the wanted dentry. That pays off while the cache can keep the dirfrag until its other entries are looked up.
It stops paying off once the tree being walked is larger than the cache:
In mdtest stat phases over 12M files, with 2 active MDS and a 16 GiB cache, each rank held about 5.3M inodes. Per rank, in 10 s:
That is about 300 lookups/s per rank, with the metadata pool read at about 34 MiB/s. The same test with 4.8M files, which fit in the cache, runs at 50–66k stat/s. Two heads of #72265 that are a week apart behave the same, and the code that does it is in main.
Fix (mds: fetch only the wanted dentry when the cache has no room for the dirfrag)
MDCache::cache_has_room_for(n) tells whether n more entries fit. It says no when:
On a lookup miss, CDir::fetch() loads only the wanted dentry (fetch_keys()) when the dirfrag does not fit. The number of entries comes from the dirfrag's fragstat, or from the directory's dirstat when a dirfrag that trimming closed comes back without its fnode.
Nothing changes while the cache has room. Readdir, and lookups with mds_dir_prefetch off, are not affected.
Test (qa/cephfs: test lookups that miss when the MDS cache is full)
TestDirPrefetchCacheFull in test_misc.py creates 1000 files in dir and drops them from the MDS cache:
Testing
qa on vstart:
IO500 with a fixed n and no stonewall, 2 active MDS, 48 tasks over 24 kernel mounts, one run per column, in kIOPS. The builds were main with #72265 and #72900, since the setup uses ceph.dir.pin.distributed.tree on the mdtest directories.
1 GiB cache, 1.34M files (about twice what the cache holds):
With the default, each rank fetched about 23 whole dirfrags and trimmed about 47k inodes per second during easy-stat, replying to about 35 lookups per second. With this PR, no whole dirfrag was fetched during the stat phases. hard-stat varied between 30 and 66 from run to run with both settings.
16 GiB cache:
Above the cache, stat lookups that miss are now bound by keyed omap reads, about 3.9k to 12k per second. Below the cache nothing changes.
The 4.8M-file runs of this PR used a revision without the trim_counter check, which cannot take effect while nothing is trimmed. All 16 GiB runs, and the default column of the 1 GiB run, were measured on a #72265 revision that predates its fix for fragments handed back when they empty. That fix matters only in delete phases.
Fixes: https://tracker.ceph.com/issues/82988
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an
xbetween the brackets:[x]. Spaces and capitalization matter when checking off items this way.Checklist
Show available Jenkins commands
jenkins test classic perfJenkins Job | Jenkins Job Definitionjenkins test crimson perfJenkins Job | Jenkins Job Definitionjenkins test signedJenkins Job | Jenkins Job Definitionjenkins test make checkJenkins Job | Jenkins Job Definitionjenkins test make check arm64Jenkins Job | Jenkins Job Definitionjenkins test submodulesJenkins Job | Jenkins Job Definitionjenkins test dashboardJenkins Job | Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job | Jenkins Job Definitionjenkins test apiJenkins Job | Jenkins Job Definitionjenkins test docsReadTheDocs | Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs | Jenkins Jobs Definitionjenkins test windowsJenkins Job | Jenkins Job Definitionjenkins test rook e2eJenkins Job | Jenkins Job DefinitionYou must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.