Repository navigation
randomness in jit_stencils.h #138061
Description
Bug report
Bug description:
While working on reproducible builds for openSUSE, I found that
our Python-3.14.0rc2 package has random variations in one of its files:
--- old//usr/src/debug/Python-3.14.0rc2/jit_stencils-x86_64-suse-linux-gnu.h
+++ new//usr/src/debug/Python-3.14.0rc2/jit_stencils-x86_64-suse-linux-gnu.h
@@ -1,4 +1,4 @@
-// 758e10176251ef35b7d8475a699dd17f8f18825312fcdd02e2e0aaa535e2898b
+// fb9d76f68f8171b81f54501d168e608c09472dc74015ec545a1e862b2bbbbf40
// $ python3.13 ./Tools/jit/build.py x86_64-suse-linux-gnu --output-dir . --pyconfig-dir . --cflags=
voidIt would be better to have all build outputs deterministic.
related: https://bugzilla.opensuse.org/show_bug.cgi?id=1244680
CPython versions tested on:
3.14
Operating systems tested on:
Linux
Linked PRs
- gh-138061: Exclude __pycache__ directory from the computed digest in the jit stencils #138131
- [3.13] gh-138061: Exclude __pycache__ directory from the computed digest in the JIT stencils (GH-138131) #142481
- [3.14] gh-138061: Exclude __pycache__ directory from the computed digest in the JIT stencils (GH-138131) #142482
Activity
ZeroIntensity commentedon Aug 22, 2025
Looking at the openSUSE report,
--disable-giland--enable-experimental-jitare combined. This does not work the way you think it does. On free-threaded builds, the JIT is disabled at runtime.That said,
jit_stencils.his autogenerated from LLVM. Is the LLVM version the same across builds?bmwiedemann commentedon Aug 22, 2025
We have 2 different builds - a normal one and
nogil. For python314, the above diff also occurs without--disable-gil.Yes, all the toolchain packages in the build env (KVM throwaway VMs) remain the same.
I wonder, if it is related to LTO. We enable that by default.
Fidget-Spinner commentedon Aug 22, 2025
The JIT doesn't use the same CFLAGS as in your configure or other script used to build CPython. You need to pass in the flags to the JIT stencil generation process directly or use
CFLAGS_JITenv var to pass it. That might be why, as I don't think standard LLVM builds are reproducible without additional flags.brandtbucher commentedon Aug 22, 2025
If it’s just the hash at the top of the file, that’s created by hashing all of the inputs to the JIT build step. This includes the target triple, whether or not this is a debug build, additional CFLAGS passed to the JIT build step, the contents of
Python/executor_cases.c.h,pyconfig.h, and anything present in theTools/jitdirectory.Are you running it twice in a row for the same build and seeing a different hash there? LLVM shouldn’t influence this hash in any way, FYI.
bmwiedemann commentedon Aug 24, 2025
When I keep the whole tree, running it twice gives the same result.
Doing more tests, I used
straceto find the list of files it opened and among these were 5.pycfiles inTools/jit/__pycache__/that did vary, maybe because those contained the mtime of.pyfiles in the header. Or because.pycfiles are just dumps of python's internal state that is not designed to be deterministic with reference-counters and such.And when I did
rm -rf Tools/jit/__pycache__ ; export PYTHONDONTWRITEBYTECODE=1ahead of the call, I now got deterministic results.It is not deterministic without the
export.One of the varying files is
_llvm.cpython-313.pycand the diff is only mtimes:brandtbucher commentedon Aug 24, 2025
Thanks for the investigation, that really helps. It’s just an oversight in the code,
.pycfiles should definitely be excluded when computing the hash.gh-138061: Exclude __pycache__ directory from the computed digest in …
pythongh-138061: Exclude __pycache__ directory from the computed dige…
pythongh-138061: Exclude __pycache__ directory from the computed dige…
pythongh-138061: Exclude __pycache__ directory from the computed dige…
[3.13] gh-138061: Exclude __pycache__ directory from the computed dig…
[3.14] gh-138061: Exclude __pycache__ directory from the computed dig…