Repository navigation
DOC, CI: Docs build is failing on CircleCI #26820
Description
See, for example, https://app.circleci.com/pipelines/github/numpy/numpy/27407/workflows/01083f16-6603-4391-a31a-d4c839924e7f/jobs/40978 or https://app.circleci.com/pipelines/github/numpy/numpy/27412/workflows/035d8506-affe-46f1-8467-249f6c6856e3/jobs/40983?invite=true#step-108-5389_21.
The error message is as follows.
Details
<snip>
Running Sphinx v7.2.6
Traceback (most recent call last):
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/config.py", line 358, in eval_config_file
exec(code, namespace) # NoQA: S102
^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/repo/doc/source/conf.py", line 65, in <module>
replace_scalar_type_names()
File "/home/circleci/repo/doc/source/conf.py", line 45, in replace_scalar_type_names
assert 'numpy._core._add_newdocs_scalars' not in sys.modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/cmd/build.py", line 293, in build_main
app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/application.py", line 211, in __init__
self.config = Config.read(self.confdir, confoverrides or {}, self.tags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/config.py", line 181, in read
namespace = eval_config_file(filename, tags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/config.py", line 371, in eval_config_file
raise ConfigError(msg % traceback.format_exc()) from exc
sphinx.errors.ConfigError: There is a programmable error in your configuration file:
Traceback (most recent call last):
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/config.py", line 358, in eval_config_file
exec(code, namespace) # NoQA: S102
^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/repo/doc/source/conf.py", line 65, in <module>
replace_scalar_type_names()
File "/home/circleci/repo/doc/source/conf.py", line 45, in replace_scalar_type_names
assert 'numpy._core._add_newdocs_scalars' not in sys.modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Configuration error:
There is a programmable error in your configuration file:
Traceback (most recent call last):
File "/home/circleci/repo/venv/lib/python3.11/site-packages/sphinx/config.py", line 358, in eval_config_file
exec(code, namespace) # NoQA: S102
^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/repo/doc/source/conf.py", line 65, in <module>
replace_scalar_type_names()
File "/home/circleci/repo/doc/source/conf.py", line 45, in replace_scalar_type_names
assert 'numpy._core._add_newdocs_scalars' not in sys.modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
make: *** [Makefile:144: html-build] Error 2
Exited with code exit status 2
My guess .circleci/config.yml needs to be updated. I'll look into it
Activity
melissawm commentedon Jul 1, 2024
I don't know exactly what's happening but here's a summary of what I observed.
The issue is in this line:
numpy/doc/source/conf.py
Line 45 in c21ac10
I tried using spin for both cases, using the exact same sphinx version etc (this is not the case in CI but I don't think it has anything to do with it.) So maybe a path or meson thing? I don't know how to debug this.
mattip commentedon Jul 2, 2024
That line came from #24634. @mtsokol any thoughts what might have changed?
mtsokol commentedon Jul 2, 2024
It only renamed
coreto_corein that PR almost a year ago.mattip commentedon Jul 2, 2024
Ahh, thanks for the clarification. Something somewhere else (docutils? sphinx?) changed and now that module is being imported, since
spin docsis now failing locally on commits where it passed previously. So it requires a careful comparison of package versions installed into a successful CI run with a failing one.mtsokol commentedon Jul 2, 2024
I checked the most recent successful CI run and a failing one and the only dependency difference was
pillowpackage (10.3.0upgraded to10.4.0).In #26825 I pinned
pillow==10.3.0and the error didn't occur.So I think the source of the issue is somewhere in https://github.com/python-pillow/Pillow/releases/tag/10.4.0
melissawm commentedon Jul 2, 2024
But why does it work with conda?
mattip commentedon Jul 2, 2024
Strange. The line (and the whole
replace_scalar_type_names()function came from #17331, in an attempt to avoid platform-dependent scalar type name differences. Now that we have cleaned much of the aliasing up, perhaps we can do that function differently or even delete it.Maybe conda is using different versions of the dependencies, which do not import the submodule
mtsokol commentedon Jul 2, 2024
Does the conda use
pillow==10.4.0? I see that10.4.0was uploaded to pip yesterday, but on conda it's added 3 hours ago: https://anaconda.org/conda-forge/pillow. I guess conda build still used10.3.0when you tested it.mtsokol commentedon Jul 2, 2024
I also managed to reproduce it locally with
pillow==10.4.0upgraded.mtsokol commentedon Jul 2, 2024
If we would like to temporarily fix it here a PR: #26825
mtsokol commentedon Jul 2, 2024
And here's a detailed explanation of what is going on:
When docs are build
sphinx-buildcommand is run which imports Sphinx. In one of its files it imports Pillow: https://github.com/sphinx-doc/sphinx/blob/d2f1acc0cade3b02d24dbdade5533405b6424b1f/sphinx/util/images.py#L15With
pillow==10.3.0Importing
ImagefromPILdidn't trigger NumPy import:With
pillow==10.4.0But in the latest release NumPy is imported:
It's due to the fact that in the latest release Pillow maintainers added type hints to ImageFilter and there's
import numpy.typing as nptinPIL/_typing.py:https://github.com/python-pillow/Pillow/blame/8c2a823e77735b4849fc921b0009220170cc82ea/src/PIL/_typing.py#L8
mtsokol commentedon Jul 2, 2024
I think one solution could be to remove
replace_scalar_type_nameshack inconf.pyand instead add an environment variable, likeUSE_PLATFORM_AGNOSTIC_SCALAR_NAMESthat is fetched in_add_newdocs_scalars.pyand based on it appropriate__name__s are attached to scalars (defaultFalse).Then in the CI command we would add it:
WDYT?
melissawm commentedon Jul 2, 2024
But that problem will then keep happening for folks using venv and building the docs locally, right? I'd prefer a solution included in
spin docs, but if that's not possible I guess we could do that.mattip commentedon Jul 2, 2024
At least locally, building without the assert and function override works cleanly for me I think.
ngoldbaum commentedon Jul 2, 2024
Arguably pillow importing numpy now when it didn’t before could be considered a bug. I don’t know if they’re aware that now numpy is an (optional) runtime dependency because of the typing change. Also good detective work @mtsokol!
melissawm commentedon Jul 2, 2024
You were right - conda was installing pillow 10.3.0
Indeed! ❤️
Separately, we should update the CircleCI workflow to use spin build/spin docs.