mwindow: Avoid holding mwindow mutex during pack and window allocations - #7443

Summary: libgit2 can deadlock when holding git_mwindow__mutex across allocations if a custom allocator, under GC pressure, runs a finalizer that re-enters libgit2 and tries to acquire the same mutex.

When libgit2 is configured with a custom allocator via GIT_OPT_SET_ALLOCATOR (such as in Ruby/Rugged bindings), memory allocations can trigger a garbage collection cycle. If the GC finalizer frees an unreachable repository, git_repository_free calls git_mwindow_put_pack, which attempts to acquire git_mwindow__mutex.

Previously, git_mwindow_get_pack held git_mwindow__mutex across git_packfile_alloc. When an allocation triggered a GC pass, the thread deadlocked on the non-recursive mutex with itself. A similar issue existed in git_mwindow_open when allocating git_mwindow structures.

  sequenceDiagram
      participant Thread as Worker Thread
      participant Mutex as git_mwindow__mutex
      participant Alloc as Custom Allocator (Ruby xmalloc)
      participant GC as Garbage Collector
  Thread->>Mutex: 1. git_mutex_lock() [ACQUIRED]
  Thread->>Alloc: 2. git_packfile_alloc() -> malloc()
  Alloc->>GC: 3. Memory pressure triggers GC
  GC->>Thread: 4. Finalizer runs: git_repository_free()
  Thread->>Mutex: 5. git_mwindow_put_pack() -> git_mutex_lock()
  Note over Thread,Mutex: 6. DEADLOCK! (Mutex already held by Step 1 on same thread)

Move git_packfile_alloc and git_mwindow allocations outside of the git_mwindow__mutex critical sections:

• In git_mwindow_get_pack, perform an optimistic cache check, release the mutex before allocating, and perform a double-checked lookup when re-acquiring the mutex.
• In git_mwindow_open, allocate git_mwindow structures outside the mutex and re-check the window list after re-acquiring the lock.
• Add regression test in tests/libgit2/pack/sharing.c with a custom allocator that triggers re-entrant repository disposal during packfile allocation.

Why it emerged recently:

• In libgit2 (latent for 12 years): git_mwindow__mutex being held across git_packfile_alloc() was introduced in commit b3b66c5 in June 2014.
• In Rugged (unmasked in 1.9.4): In Rugged <= 1.9.0, Rugged::Repository used untyped Data_Wrap_Struct where finalizers were deferred. In Rugged 1.9.4 (PR Use TypedData instead of Data rugged#998), Rugged converted all classes to TypedData_Wrap_Struct with RUBY_TYPED_FREE_IMMEDIATELY [ref for repo], allowing Ruby's GC to run rb_git_repo__free synchronously inside xmalloc allocations and directly triggering the lock re-entrancy deadlock.

I'm not sure if there are other places in the code where this can happen as well, but I suspect there are others. In general, we shouldn't do an allocation within a mutex, but especially since the allocator can do more than allocate.