Limit qemu-img execution arenas

juliakreger · juliakreger · commit ee8118423b88 · 2021-06-01T12:26:38.000-07:00
qemu-img attempts to launch multiple threads by default *and* attempts to have multiple memory allocation arenas to operate from. While multithreading can be good for performance, this pattern and the memory footprint for process launch and dependencies can turn the memory footprint for a cirros image conversion (16MB) into 1.2GB of memory being asked for by the qemu-img tool. In order to limit this impact, as the default number of arenas is governed by the number of CPUs times the number 8, it seems reasonable to lower this to a more reasonable number which also helps keep our possible memory footprint from being exceeded. NOTE: This change is largely different than the original change as an intermediate change converted write_image.sh to python. As it is unlikely for us to backport the intermediate change, it is logical for us to just modify the original script. Otherwise the release note is ultimately what is backported for release note tooling continutity. Change-Id: I71a28ec59ec31c691205eb34d9fcab63a2ccb682 Story: 2008928 Task: 42528 (cherry picked from commit 9e4c705) (cherry picked from commit 9c20cca)
diff --git a/ironic_python_agent/shell/write_image.sh b/ironic_python_agent/shell/write_image.sh
@@ -49,6 +49,15 @@ log "Imaging $IMAGEFILE to $DEVICE"
 
 # limit the memory usage for qemu-img to 2 GiB
 ulimit -v 2097152
+# NOTE(TheJulia): qemu-img uses multiple threads by default and in
+# cross-thread memory allocation lock conflicts, glibc will ultimately
+# attempt to provide it with an additional arena to allocate from, however
+# the running default, when not overridden is 8 * ncpu * the footprint, which
+# very quickly exceeds the ulimit. This is most observable on CI systems where
+# cross-vcpu thread locking can result in a conflict that wouldn't normally be
+# as likely on physical hardware.
+# See discussion on https://bugzilla.redhat.com/show_bug.cgi?id=1892773
+export MALLOC_ARENA_MAX=3
 qemu-img convert -t directsync -O host_device $IMAGEFILE $DEVICE
 sync
 
diff --git a/releasenotes/notes/limit-qemu-img-malloc-arena-025ed84115481eae.yaml b/releasenotes/notes/limit-qemu-img-malloc-arena-025ed84115481eae.yaml
@@ -0,0 +1,7 @@
+---
+fixes:
+  - |
+    Fixes failures with disk image conversions which result in memory
+    allocation or input/output errors due to memory limitations by limiting
+    the number of available memory allocation pools to a non-dynamic
+    reasonable number which should not exceed the available system memory.
