py/dynruntime.mk: Let natmods be built with Clang.#19308
Draft
agatti wants to merge 4 commits into
Draft
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19308 +/- ##
=======================================
Coverage 98.47% 98.47%
=======================================
Files 176 176
Lines 22845 22845
=======================================
Hits 22497 22497
Misses 348 348 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Code size report: |
Contributor
Author
|
Still that's not enough, as then link fails even with all the symbols being present: make CC=clang ARCH=x64 V=1
/bin/mkdir -p build/
GEN build/deflate_x64.config.h
python3 ../../../tools/mpy_ld.py '-vvv' --arch x64 --preprocess -o build/deflate_x64.config.h deflate.c
CC deflate.c
clang -I. -I../../.. -std=c99 -Os -Wall -Werror -DNDEBUG -DNO_QSTR -DMICROPY_ENABLE_DYNRUNTIME -DMP_CONFIGFILE='<build/deflate_x64.config.h>' -fpic -fno-common -U_FORTIFY_SOURCE -fno-stack-protector -DMICROPY_FLOAT_IMPL=MICROPY_FLOAT_IMPL_DOUBLE -o build/deflate.o -c deflate.c
LINK build/deflate.o
python3 ../../../tools/mpy_ld.py '-vvv' --arch x64 --qstrs build/deflate_x64.config.h -l/usr/lib/gcc/x86_64-linux-gnu/13/libgcc.a -l/usr/lib/x86_64-linux-gnu/libm.a -l/usr/lib/x86_64-linux-gnu/libc.a -o build/deflate_x64.native.mpy build/deflate.o
qstr vals: DeflateIO, GZIP, RAW, ZLIB, deflate
Loading /usr/lib/gcc/x86_64-linux-gnu/13/libgcc.a
Loading /usr/lib/x86_64-linux-gnu/libm-2.39.a
Loading /usr/lib/x86_64-linux-gnu/libmvec.a
Loading /usr/lib/x86_64-linux-gnu/libc.a
Skippping weak dependency: _dl_find_object_init
...
Skippping weak dependency: _nl_C_LC_MONETARY
using /usr/lib/x86_64-linux-gnu/libc.a:memset.o
using /usr/lib/x86_64-linux-gnu/libc.a:memset-evex-unaligned-erms.o
using /usr/lib/x86_64-linux-gnu/libc.a:memset-sse2-unaligned-erms.o
using /usr/lib/x86_64-linux-gnu/libc.a:memset-erms.o
using /usr/lib/x86_64-linux-gnu/libc.a:dl-support.o
LinkError: /usr/lib/x86_64-linux-gnu/libc.a:dl-support.o: .data non-empty
make: *** [../../../py/dynruntime.mk:255: build/deflate_x64.native.mpy] Error 1Edit: [1] diff --git i/py/dynruntime.mk w/py/dynruntime.mk
index 26f2fc720..4c5b9655f 100644
--- i/py/dynruntime.mk
+++ w/py/dynruntime.mk
@@ -148,7 +148,7 @@ else
$(error architecture '$(ARCH)' not supported)
endif
-ifeq ($(firstword $(shell $(CC) --version)),clang)
+ifeq ($(findstring clang,$(shell $(CC) --version)),clang)
CROSS =
endif
@@ -199,6 +199,10 @@ LIBM_PATH := $(PICOLIBC_ROOT)/$(PICOLIBC_ARCH)/$(PICOLIBC_ABI)/$(LIBM_NAME)
endif
endif
MPY_LD_FLAGS += $(addprefix -l, $(LIBGCC_PATH) $(LIBM_PATH))
+ifeq ($(findstring clang,$(shell $(CC) --version)),clang)
+LIBC_PATH := $(realpath $(shell $(CROSS)$(CC) $(CFLAGS) --print-file-name=libc.a))
+MPY_LD_FLAGS += $(addprefix -l, $(LIBC_PATH))
+endif
endif
ifneq ($(MPY_EXTERN_SYM_FILE),)
MPY_LD_FLAGS += --externs "$(realpath $(MPY_EXTERN_SYM_FILE))" |
This commit modifies the build rules for native modules in order to remove the dependence on GCC for creating native MPY files. Whilst the Unix port of MicroPython can be built with Clang by overriding the `CC` variable, natmods require a bit more work. GCC builds compilers that are tailored for a single architecture, but Clang takes the opposite approach, so a single binary may target more than one architecture. Architecture selection is, by definition, not compatible between those two compilers. These changes attempt to make things easier to handle when using Clang. Native modules can now be built with something like this: make CC=clang ARCH=<arch> \ CFLAGS_EXTRA='--target=<clang-target-arch>' \ LDFLAGS_EXTRA='--target=<clang-target-arch>' So, for example building an x86 native module the command line will look something like this: make CC=clang ARCH=x86 \ CFLAGS_EXTRA='--target=i686-unknown-linux-gnu' \ LDFLAGS_EXTRA='--target=i686-unknown-linux-gnu' Clang and GCC, however, have different tolerances for deviations from the chosen C standard. Whilst GCC doesn't really mind whether a typedef is defined multiple times as long as it is defined to the same value, Clang does raise a warning which is then interpreted as an error. Unfortunately #ifdef/#ifndef does not work with typedefs, and the way native modules are built meant that `py/mpconfig.h` would first include the native module's generated configuration file and then proceed with the rest of the configuration. However, both files attempt to provide aliases for both `mp_int_t` and `mp_uint_t`, and that doesn't really work. Thus, the only sane way to work around it is to rely on the presence of a definition that indicates that `mp_int_t` and `mp_uint_t` are already there to begin with, letting builds proceed on both GCC and Clang. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit fixes building the `deflate` module using Clang rather than using GCC. The Clang standard library (libc) implementation of `memset` depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit fixes building the `framebuf` module using Clang rather than using GCC. The Clang standard library (libc) implementation of `memset` depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit fixes building the `re` module using Clang rather than using GCC. The Clang standard library (libc) implementation of `memset` depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Contributor
Author
|
Even more fun, on x86 at least, clang will put 
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR modifies the build rules for native modules in order to remove the dependence on GCC for creating native MPY files.
Whilst the Unix port of MicroPython can be built with Clang by overriding the
CCvariable, natmods require a bit more work. GCC builds compilers that are tailored for a single architecture, but Clang takes the opposite approach, so a single binary may target more than one architecture. Architecture selection is, by definition, not compatible between those two compilers.These changes attempt to make things easier to handle when using Clang. Native modules can now be built with something like this:
make CC=clang ARCH=<arch> CFLAGS_EXTRA='--target=<clang-target-arch>' LDFLAGS_EXTRA='--target=<clang-target-arch>'. So, for example building an x86 native module the command line will look something like this:make CC=clang ARCH=x86 CFLAGS_EXTRA='--target=i686-unknown-linux-gnu' LDFLAGS_EXTRA='--target=i686-unknown-linux-gnu'.Clang and GCC, however, have different tolerances for deviations from the chosen C standard. Whilst GCC doesn't really mind whether a typedef is defined multiple times as long as it is defined to the same value, Clang does raise a warning which is then interpreted as an error.
Unfortunately #ifdef/#ifndef does not work with typedefs, and the way native modules are built meant that
py/mpconfig.hwould first include the native module's generated configuration file and then proceed with the rest of the configuration. However, both files attempt to provide aliases for bothmp_int_tandmp_uint_t, and that doesn't really work. Thus, the only sane way to work around it is to rely on the presence of a definition that indicates thatmp_int_tandmp_uint_tare already there to begin with, letting builds proceed on both GCC and Clang.Testing
All natmods in
examples/natmodwere attempted to be built for x86 using the command line mentioned in the summary section. Results are as follows:btree: fails to compile because__bt_close,__bt_sync, andbt_metado not have a prototypedeflate:fails to link becausebuilds on both x86 and x64memsetis not definedfeatures0: buildsfeatures1: buildsfeatures2: buildsfeatures3: buildsfeatures4: buildsframebuf:fails to link becausebuilds on both x86 and x64memsetis not definedheapq: buildsrandom: buildsre:fails to link becausebuilds on both x86 and x64.memsetis not definedImporting
features0in a Clang-built x86 interpreter, though, crashes due to "something" happening when callingmp_call_function_0. The same happens on an x86 interpreter built with GCC, and on x64 too. Fun for the whole architecture family.The same procedure was done successfully without overriding
CC,CFLAGS_EXTRA, andLDFLAGS_EXTRA, still building withARCH=x86. Then natmods were built forxtensawinbut specifyingCROSS=xtensa-esp32s3-elf-to make sure the new compiler was picked up correctly.Trade-offs and Alternatives
None I can see, although the crashes and link failures need to be sorted out first.
Generative AI
I did not use generative AI tools when creating this PR.
This is marked as draft due to the obvious shortcomings of the PR in its current state. The PR was still submitted in the hope that somebody more experienced with Clang will sort the issues out before I get to that.