Skip to content

Commit ea95fb6

Browse files
ShillakerShillaker
committed
Crossenv debugging
1 parent a7a7468 commit ea95fb6

6 files changed

Lines changed: 90 additions & 40 deletions

File tree

‎.gitmodules‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,5 @@
77
url = https://github.com/Shillaker/numpy
88
[submodule "third-party/crossenv"]
99
path = third-party/crossenv
10-
url = git@github.com:benfogle/crossenv.git
10+
url = https://github.com/Shillaker/crossenv
11+
branch = faasm

‎README.md‎

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,12 @@ provided we don't need them.
4343
At the end of the CPython build, it will print out a list of which modules have
4444
been successfully built and which have failed.
4545

46-
## Cross-compiling modules
46+
## Cross-compilation set-up
4747

4848
Setuptools and distutils both interrogate the Python system environment during
4949
the build process. This makes it quite difficult to cross-compile libraries, so
5050
we use [crossenv](https://github.com/benfogle/crossenv).
5151

52-
### Crossenv set-up
53-
5452
To set up crossenv for the first time:
5553

5654
```
@@ -63,9 +61,26 @@ You can then activate with:
6361
. cross_venv/bin/activate
6462
```
6563

66-
### Building modules
64+
From inside the virtual environment, you can inspect the set-up with:
65+
66+
```
67+
python sanity_check.py | less
68+
```
69+
70+
### Changing the crossenv environment
71+
72+
Note that crossenv picks up the cross-compilation environment from the CPython
73+
build. Therefore, to make changes:
74+
75+
- Modify the CPython build (see `tasks.py`)
76+
- Rerun the CPython build (`inv cpython.py --clean`)
77+
- Rebuild the crossenv (`./crossenv_setup.sh`)
78+
- Enter the crossenv and inspect the environment with `sanity_check.py`
6779

68-
With the crossenv activated, we can build modules with normal `pip`:
80+
## Modules
81+
82+
With the crossenv activated, we can build modules with normal `pip`,
83+
e.g.
6984

7085
```
7186
cd third-party/numpy
@@ -78,24 +93,11 @@ To get the pip logs add the `--log` argument.
7893
pip install . --log /tmp/pip.log
7994
```
8095

81-
### Crossenv environment issues
82-
83-
Note that crossenv should pick up _most_ of the required cross-compilation
84-
environment from the CPython build artifacts. The bits that it doesn't pick up
85-
will be set in `crossenv.sh`, so check _both_ when debugging issues.
86-
87-
The config seen by crossenv will be echoed in
88-
`cross_venv/lib/_sysconfigdata_wasi_.py`.
89-
90-
Because crossenv picks up its info from the CPython build, any changes, to the
91-
cross-compile environment must be replicated in the CPython build.
92-
93-
### Runtime files
96+
## Runtime files
9497

9598
CPython needs to access certain files at runtime. These can be put in place with
9699
the following:
97100

98101
```
99102
inv runtime
100103
```
101-

‎crossenv_setup.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
import logging
2+
from os.path import join, abspath, dirname, exists
3+
from shutil import rmtree
4+
5+
from crossenv import CrossEnvBuilder
6+
7+
THIS_DIR = dirname(abspath(__file__))
8+
9+
WASM_CPYTHON = join(THIS_DIR, "third-party", "cpython", "install", "wasm",
10+
"bin", "python3.8")
11+
12+
CROSSENV_VENV_DIR = join(THIS_DIR, "cross_venv")
13+
14+
# TODO - should this be the actual sysroot, or the Python install dir?
15+
CROSSENV_SYSROOT = "/usr/local/faasm/llvm-sysroot"
16+
17+
18+
def main():
19+
# Clean existing crossenv virtual environment
20+
if exists(CROSSENV_VENV_DIR):
21+
print("Removing existing crossenv at {}".format(CROSSENV_VENV_DIR))
22+
rmtree(CROSSENV_VENV_DIR)
23+
24+
# Set logging level
25+
level = logging.DEBUG
26+
logging.basicConfig(level=level, format='%(levelname)s: %(message)s')
27+
28+
# Invoke the cross-env builder
29+
print("Building crossenv at {}".format(CROSSENV_VENV_DIR))
30+
builder = CrossEnvBuilder(
31+
host_python=WASM_CPYTHON,
32+
host_sysroot=CROSSENV_SYSROOT,
33+
)
34+
builder.create(CROSSENV_VENV_DIR)
35+
36+
37+
if __name__ == "__main__":
38+
main()

crossenv.sh renamed to crossenv_setup.sh

Lines changed: 6 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -6,31 +6,18 @@ THIS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
66

77
# NOTE - all the commands must use the specially-installed python
88
# on the build machine.
9-
109
BUILD_PYTHON_BIN=/usr/local/faasm/python3.8/bin
1110
BUILD_PYTHON=${BUILD_PYTHON_BIN}/python3.8
1211
BUILD_PIP=${BUILD_PYTHON_BIN}/pip3.8
1312

14-
WASM_SYSROOT=/usr/local/faasm/llvm-sysroot
15-
WASM_CPYTHON=${THIS_DIR}/third-party/cpython/install/wasm/bin/python3.8
16-
17-
CROSSENV_VENV_DIR=${THIS_DIR}/cross_venv
18-
CROSSENV_SRC_DIR=${THIS_DIR}/third-party/crossenv
19-
20-
# Clean existing crossenv virtual environment
21-
if [ -d "$CROSSENV_VENV_DIR" ]; then
22-
rm -r ${CROSSENV_VENV_DIR}
23-
fi
24-
2513
# Install the crossenv module in dev mode
14+
CROSSENV_SRC_DIR=${THIS_DIR}/third-party/crossenv
15+
echo "Installing crossenv from ${CROSSENV_SRC_DIR}"
2616
pushd ${CROSSENV_SRC_DIR} >> /dev/null
2717
${BUILD_PIP} install -e .
2818
popd >> /dev/null
2919

30-
# Set up crossenv
31-
${BUILD_PYTHON} -m crossenv \
32-
${WASM_CPYTHON} \
33-
${CROSSENV_VENV_DIR} \
34-
-vvv \
35-
--sysroot=${WASM_SYSROOT}
36-
20+
# Run the set-up script
21+
pushd ${THIS_DIR} >> /dev/null
22+
${BUILD_PYTHON} crossenv_setup.py
23+
popd >> /dev/null

‎sanity_check.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import distutils.ccompiler
2+
import sysconfig
3+
from pprint import pprint
4+
5+
6+
def main():
7+
# Sysconfig checks
8+
print("Platform: {}".format(sysconfig.get_platform()))
9+
print("Python version: {}".format(sysconfig.get_python_version()))
10+
print("Current installation scheme: {}".format(
11+
sysconfig._get_default_scheme()))
12+
print("Paths")
13+
pprint(sysconfig.get_paths())
14+
print("Variables")
15+
pprint(sysconfig.get_config_vars())
16+
17+
# Distutils checks
18+
print("Compiler: {}".format(distutils.ccompiler.get_default_compiler()))
19+
20+
21+
if __name__ == "__main__":
22+
main()

‎third-party/crossenv‎

0 commit comments

Comments
 (0)