git: worktree, optimize infiles function for very large repos#1853
git: worktree, optimize infiles function for very large repos#1853pjbgf merged 3 commits intogo-git:releases/v5.xfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR optimizes the inFiles function for very large repositories by replacing a linear search (O(n*m) complexity) with a map-based lookup (O(n+m) complexity). The optimization is particularly beneficial when dealing with large numbers of changes and file filters during git reset operations.
Changes:
- Replaced slice-based linear search in
inFileswith O(1) map lookup - Added
buildFilePathMaphelper function to create file path maps with cleaned paths - Pre-allocated
removedFilesslice capacity inresetIndexfor better memory efficiency
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } | ||
|
|
||
| func inFiles(files []string, v string) bool { | ||
| // inFiles checks if the given file is in the list of files. The incoming filepaths in files should be cleaned before calling this function. |
There was a problem hiding this comment.
The comment states "The incoming filepaths in files should be cleaned before calling this function", but this is misleading. The function already cleans the input parameter v on line 442, and the map parameter files is already built with cleaned paths by buildFilePathMap. Consider revising the comment to clarify that the map is expected to contain pre-cleaned paths (which is handled by buildFilePathMap), and that the function will clean the lookup key v internally.
| // inFiles checks if the given file is in the list of files. The incoming filepaths in files should be cleaned before calling this function. | |
| // inFiles checks if the given file is in the list of files. The files map is expected | |
| // to contain pre-cleaned paths (as produced by buildFilePathMap), and this function | |
| // will clean the lookup key v before checking for membership. |
|
Is it possible to have a benchmark or is there an existing benchmark that get better when using this PR? |
@cedric-appdirect I agree benchmarking performance changes is a good practice. On this case specifically I'm slighly less worried due to the change itself. That being said, I'm planning to have some level of performance regression tests into CI in the near future. |
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | `v5.16.5` -> `v5.17.0` |  |  | --- ### Release Notes <details> <summary>go-git/go-git (github.com/go-git/go-git/v5)</summary> ### [`v5.17.0`](https://github.com/go-git/go-git/releases/tag/v5.17.0) [Compare Source](go-git/go-git@v5.16.5...v5.17.0) #### What's Changed - build: Update module github.com/go-git/go-git/v5 to v5.16.5 \[SECURITY] (releases/v5.x) by [@​go-git-renovate](https://github.com/go-git-renovate)\[bot] in [#​1839](go-git/go-git#1839) - git: worktree, optimize infiles function for very large repos by [@​k-anshul](https://github.com/k-anshul) in [#​1853](go-git/go-git#1853) - git: Add strict checks for supported extensions by [@​pjbgf](https://github.com/pjbgf) in [#​1861](go-git/go-git#1861) - backport, git: Improve Status() speed with new index.ModTime check by [@​cedric-appdirect](https://github.com/cedric-appdirect) in [#​1862](go-git/go-git#1862) - storage: filesystem, Avoid overwriting loose obj files by [@​pjbgf](https://github.com/pjbgf) in [#​1864](go-git/go-git#1864) **Full Changelog**: <go-git/go-git@v5.16.5...v5.17.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM ( * 0-3 * * * ) (UTC), Automerge - Between 12:00 AM and 03:59 AM ( * 0-3 * * * ) (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My41LjAiLCJ1cGRhdGVkSW5WZXIiOiI0My41LjAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbIktpbmQvRGVwZW5kZW5jeVVwZGF0ZSIsInJ1bi1lbmQtdG8tZW5kLXRlc3RzIl19--> Reviewed-on: https://code.forgejo.org/forgejo/runner/pulls/1410 Reviewed-by: Mathieu Fenniak <mfenniak@noreply.code.forgejo.org> Co-authored-by: Renovate Bot <bot@kriese.eu> Co-committed-by: Renovate Bot <bot@kriese.eu>
This PR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | require | minor | `v5.16.5` → `v5.17.0` | [](https://securityscorecards.dev/viewer/?uri=github.com/go-git/go-git) | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>go-git/go-git (github.com/go-git/go-git/v5)</summary> ### [`v5.17.0`](https://github.com/go-git/go-git/releases/tag/v5.17.0) [Compare Source](go-git/go-git@v5.16.5...v5.17.0) #### What's Changed - build: Update module github.com/go-git/go-git/v5 to v5.16.5 \[SECURITY] (releases/v5.x) by [@​go-git-renovate](https://github.com/go-git-renovate)\[bot] in [#​1839](go-git/go-git#1839) - git: worktree, optimize infiles function for very large repos by [@​k-anshul](https://github.com/k-anshul) in [#​1853](go-git/go-git#1853) - git: Add strict checks for supported extensions by [@​pjbgf](https://github.com/pjbgf) in [#​1861](go-git/go-git#1861) - backport, git: Improve Status() speed with new index.ModTime check by [@​cedric-appdirect](https://github.com/cedric-appdirect) in [#​1862](go-git/go-git#1862) - storage: filesystem, Avoid overwriting loose obj files by [@​pjbgf](https://github.com/pjbgf) in [#​1864](go-git/go-git#1864) **Full Changelog**: <go-git/go-git@v5.16.5...v5.17.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At 12:00 AM through 04:59 AM and 10:00 PM through 11:59 PM, Monday through Friday ( * 0-4,22-23 * * 1-5 ), Only on Sunday and Saturday ( * * * * 0,6 ) (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xNS4yIiwidXBkYXRlZEluVmVyIjoiNDMuMTUuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiS2luZC9EZXBlbmRlbmNpZXMiXX0=--> Reviewed-on: https://altlinux.space/stapler/stplr/pulls/333 Co-authored-by: Renovate Bot <stapler-helper-bot@noreply.altlinux.space> Co-committed-by: Renovate Bot <stapler-helper-bot@noreply.altlinux.space>
…0/forgejo) (#11898) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | `v5.16.5` → `v5.17.1` |  |  | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/2779) for more information. --- ### go-git missing validation decoding Index v4 files leads to panic [CVE-2026-33762](https://nvd.nist.gov/vuln/detail/CVE-2026-33762) / [GHSA-gm2x-2g9h-ccm8](GHSA-gm2x-2g9h-ccm8) <details> <summary>More information</summary> #### Details ##### Impact `go-git`’s index decoder for format version 4 fails to validate the path name prefix length before applying it to the previously decoded path name. A maliciously crafted index file can trigger an out-of-bounds slice operation, resulting in a runtime panic during normal index parsing. This issue only affects Git index format version 4. Earlier formats (`go-git` supports only `v2` and `v3`) are not vulnerable to this issue. An attacker able to supply a crafted `.git/index` file can cause applications using go-git to panic while reading the index. If the application does not recover from panics, this results in process termination, leading to a denial-of-service (DoS) condition. Exploitation requires the ability to modify or inject a Git index file within the local repository in disk. This typically implies write access to the `.git` directory. ##### Patches Users should upgrade to `v5.17.1`, or the latest `v6` [pseudo-version](https://go.dev/ref/mod#pseudo-versions), in order to mitigate this vulnerability. ##### Credit go-git maintainers thank @​kq5y for finding and reporting this issue privately to the `go-git` project. #### Severity - CVSS Score: 2.8 / 10 (Low) - Vector String: `CVSS:3.1/AV:L/AC:L/PR:L/UI:R/S:U/C:N/I:N/A:L` #### References - [https://github.com/go-git/go-git/security/advisories/GHSA-gm2x-2g9h-ccm8](https://github.com/go-git/go-git/security/advisories/GHSA-gm2x-2g9h-ccm8) - [https://github.com/go-git/go-git](https://github.com/go-git/go-git) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-gm2x-2g9h-ccm8) and the [GitHub Advisory Database](https://github.com/github/advisory-database) ([CC-BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### go-git: Maliciously crafted idx file can cause asymmetric memory consumption [CVE-2026-34165](https://nvd.nist.gov/vuln/detail/CVE-2026-34165) / [GHSA-jhf3-xxhw-2wpp](GHSA-jhf3-xxhw-2wpp) <details> <summary>More information</summary> #### Details ##### Impact A vulnerability has been identified in which a maliciously crafted `.idx` file can cause asymmetric memory consumption, potentially exhausting available memory and resulting in a Denial of Service (DoS) condition. Exploitation requires write access to the local repository's `.git` directory, it order to create or alter existing `.idx` files. ##### Patches Users should upgrade to `v5.17.1`, or the latest `v6` [pseudo-version](https://go.dev/ref/mod#pseudo-versions), in order to mitigate this vulnerability. ##### Credit The go-git maintainers thank @​kq5y for finding and reporting this issue privately to the `go-git` project. #### Severity - CVSS Score: 5.0 / 10 (Medium) - Vector String: `CVSS:3.1/AV:L/AC:L/PR:L/UI:R/S:U/C:N/I:N/A:H` #### References - [https://github.com/go-git/go-git/security/advisories/GHSA-jhf3-xxhw-2wpp](https://github.com/go-git/go-git/security/advisories/GHSA-jhf3-xxhw-2wpp) - [https://github.com/go-git/go-git](https://github.com/go-git/go-git) - [https://github.com/go-git/go-git/releases/tag/v5.17.1](https://github.com/go-git/go-git/releases/tag/v5.17.1) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-jhf3-xxhw-2wpp) and the [GitHub Advisory Database](https://github.com/github/advisory-database) ([CC-BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### Release Notes <details> <summary>go-git/go-git (github.com/go-git/go-git/v5)</summary> ### [`v5.17.1`](https://github.com/go-git/go-git/releases/tag/v5.17.1) [Compare Source](go-git/go-git@v5.17.0...v5.17.1) #### What's Changed - build: Update module github.com/cloudflare/circl to v1.6.3 \[SECURITY] (releases/v5.x) by [@​go-git-renovate](https://github.com/go-git-renovate)\[bot] in [#​1930](go-git/go-git#1930) - \[v5] plumbing: format/index, Improve v4 entry name validation by [@​pjbgf](https://github.com/pjbgf) in [#​1935](go-git/go-git#1935) - \[v5] plumbing: format/idxfile, Fix version and fanout checks by [@​pjbgf](https://github.com/pjbgf) in [#​1937](go-git/go-git#1937) **Full Changelog**: <go-git/go-git@v5.17.0...v5.17.1> ### [`v5.17.0`](https://github.com/go-git/go-git/releases/tag/v5.17.0) [Compare Source](go-git/go-git@v5.16.5...v5.17.0) #### What's Changed - build: Update module github.com/go-git/go-git/v5 to v5.16.5 \[SECURITY] (releases/v5.x) by [@​go-git-renovate](https://github.com/go-git-renovate)\[bot] in [#​1839](go-git/go-git#1839) - git: worktree, optimize infiles function for very large repos by [@​k-anshul](https://github.com/k-anshul) in [#​1853](go-git/go-git#1853) - git: Add strict checks for supported extensions by [@​pjbgf](https://github.com/pjbgf) in [#​1861](go-git/go-git#1861) - backport, git: Improve Status() speed with new index.ModTime check by [@​cedric-appdirect](https://github.com/cedric-appdirect) in [#​1862](go-git/go-git#1862) - storage: filesystem, Avoid overwriting loose obj files by [@​pjbgf](https://github.com/pjbgf) in [#​1864](go-git/go-git#1864) **Full Changelog**: <go-git/go-git@v5.16.5...v5.17.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - Between 12:00 AM and 03:59 AM ( * 0-3 * * * ) (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My45OS4xIiwidXBkYXRlZEluVmVyIjoiNDMuOTkuMSIsInRhcmdldEJyYW5jaCI6InYxMS4wL2Zvcmdlam8iLCJsYWJlbHMiOlsiZGVwZW5kZW5jeS11cGdyYWRlIiwidGVzdC9ub3QtbmVlZGVkIl19--> Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11898 Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org> Co-authored-by: Renovate Bot <bot@kriese.eu> Co-committed-by: Renovate Bot <bot@kriese.eu>
…v14.0/forgejo) (#11899) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | `v5.16.5` → `v5.17.1` |  |  | --- ### go-git missing validation decoding Index v4 files leads to panic [CVE-2026-33762](https://nvd.nist.gov/vuln/detail/CVE-2026-33762) / [GHSA-gm2x-2g9h-ccm8](GHSA-gm2x-2g9h-ccm8) <details> <summary>More information</summary> #### Details ##### Impact `go-git`’s index decoder for format version 4 fails to validate the path name prefix length before applying it to the previously decoded path name. A maliciously crafted index file can trigger an out-of-bounds slice operation, resulting in a runtime panic during normal index parsing. This issue only affects Git index format version 4. Earlier formats (`go-git` supports only `v2` and `v3`) are not vulnerable to this issue. An attacker able to supply a crafted `.git/index` file can cause applications using go-git to panic while reading the index. If the application does not recover from panics, this results in process termination, leading to a denial-of-service (DoS) condition. Exploitation requires the ability to modify or inject a Git index file within the local repository in disk. This typically implies write access to the `.git` directory. ##### Patches Users should upgrade to `v5.17.1`, or the latest `v6` [pseudo-version](https://go.dev/ref/mod#pseudo-versions), in order to mitigate this vulnerability. ##### Credit go-git maintainers thank @​kq5y for finding and reporting this issue privately to the `go-git` project. #### Severity - CVSS Score: 2.8 / 10 (Low) - Vector String: `CVSS:3.1/AV:L/AC:L/PR:L/UI:R/S:U/C:N/I:N/A:L` #### References - [https://github.com/go-git/go-git/security/advisories/GHSA-gm2x-2g9h-ccm8](https://github.com/go-git/go-git/security/advisories/GHSA-gm2x-2g9h-ccm8) - [https://github.com/go-git/go-git](https://github.com/go-git/go-git) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-gm2x-2g9h-ccm8) and the [GitHub Advisory Database](https://github.com/github/advisory-database) ([CC-BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### go-git: Maliciously crafted idx file can cause asymmetric memory consumption [CVE-2026-34165](https://nvd.nist.gov/vuln/detail/CVE-2026-34165) / [GHSA-jhf3-xxhw-2wpp](GHSA-jhf3-xxhw-2wpp) <details> <summary>More information</summary> #### Details ##### Impact A vulnerability has been identified in which a maliciously crafted `.idx` file can cause asymmetric memory consumption, potentially exhausting available memory and resulting in a Denial of Service (DoS) condition. Exploitation requires write access to the local repository's `.git` directory, it order to create or alter existing `.idx` files. ##### Patches Users should upgrade to `v5.17.1`, or the latest `v6` [pseudo-version](https://go.dev/ref/mod#pseudo-versions), in order to mitigate this vulnerability. ##### Credit The go-git maintainers thank @​kq5y for finding and reporting this issue privately to the `go-git` project. #### Severity - CVSS Score: 5.0 / 10 (Medium) - Vector String: `CVSS:3.1/AV:L/AC:L/PR:L/UI:R/S:U/C:N/I:N/A:H` #### References - [https://github.com/go-git/go-git/security/advisories/GHSA-jhf3-xxhw-2wpp](https://github.com/go-git/go-git/security/advisories/GHSA-jhf3-xxhw-2wpp) - [https://github.com/go-git/go-git](https://github.com/go-git/go-git) - [https://github.com/go-git/go-git/releases/tag/v5.17.1](https://github.com/go-git/go-git/releases/tag/v5.17.1) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-jhf3-xxhw-2wpp) and the [GitHub Advisory Database](https://github.com/github/advisory-database) ([CC-BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### Release Notes <details> <summary>go-git/go-git (github.com/go-git/go-git/v5)</summary> ### [`v5.17.1`](https://github.com/go-git/go-git/releases/tag/v5.17.1) [Compare Source](go-git/go-git@v5.17.0...v5.17.1) #### What's Changed - build: Update module github.com/cloudflare/circl to v1.6.3 \[SECURITY] (releases/v5.x) by [@​go-git-renovate](https://github.com/go-git-renovate)\[bot] in [#​1930](go-git/go-git#1930) - \[v5] plumbing: format/index, Improve v4 entry name validation by [@​pjbgf](https://github.com/pjbgf) in [#​1935](go-git/go-git#1935) - \[v5] plumbing: format/idxfile, Fix version and fanout checks by [@​pjbgf](https://github.com/pjbgf) in [#​1937](go-git/go-git#1937) **Full Changelog**: <go-git/go-git@v5.17.0...v5.17.1> ### [`v5.17.0`](https://github.com/go-git/go-git/releases/tag/v5.17.0) [Compare Source](go-git/go-git@v5.16.5...v5.17.0) #### What's Changed - build: Update module github.com/go-git/go-git/v5 to v5.16.5 \[SECURITY] (releases/v5.x) by [@​go-git-renovate](https://github.com/go-git-renovate)\[bot] in [#​1839](go-git/go-git#1839) - git: worktree, optimize infiles function for very large repos by [@​k-anshul](https://github.com/k-anshul) in [#​1853](go-git/go-git#1853) - git: Add strict checks for supported extensions by [@​pjbgf](https://github.com/pjbgf) in [#​1861](go-git/go-git#1861) - backport, git: Improve Status() speed with new index.ModTime check by [@​cedric-appdirect](https://github.com/cedric-appdirect) in [#​1862](go-git/go-git#1862) - storage: filesystem, Avoid overwriting loose obj files by [@​pjbgf](https://github.com/pjbgf) in [#​1864](go-git/go-git#1864) **Full Changelog**: <go-git/go-git@v5.16.5...v5.17.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - Between 12:00 AM and 03:59 AM ( * 0-3 * * * ) (UTC). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My45OS4xIiwidXBkYXRlZEluVmVyIjoiNDMuOTkuMSIsInRhcmdldEJyYW5jaCI6InYxNC4wL2Zvcmdlam8iLCJsYWJlbHMiOlsiZGVwZW5kZW5jeS11cGdyYWRlIiwidGVzdC9ub3QtbmVlZGVkIl19--> Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/11899 Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org> Co-authored-by: Renovate Bot <bot@kriese.eu> Co-committed-by: Renovate Bot <bot@kriese.eu>
Backport for #1852.
Closes #1845.