Skip to content

CoreCLR msbuild (xplat) support. Initial compilable version. #518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
filmor merged 12 commits into from
Sep 21, 2017
Merged

CoreCLR msbuild (xplat) support. Initial compilable version. #518

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0abfde0
Full featured xplat build.
Jul 28, 2017
8b0b399
.Net 45 TargetingPack System.XML.dll naming fix. (For xplat linux bui…
Aug 18, 2017
3f02c12
Setup.py --xplat option refactored. Travis-ci build matrix extended.
Aug 18, 2017
78d5d46
AppVeyor matrix extended, xplat build added.
Aug 19, 2017
2dc1b47
appveyor.yml yaml syntax fix.
Aug 20, 2017
665b9d7
NUnit dependency upgraded to 3.7. Changelog improved.
Aug 20, 2017
2ac981b
Build order improvement.
Aug 22, 2017
ed48618
Mono builds now can be build on Windows.
Sep 6, 2017
966f035
PYTHONNET_PY3_VERSION, PYTHONNET_PY2_VERSION build related environmen…
Sep 7, 2017
f4a6608
Small compile fixes.
Sep 7, 2017
a89685e
PYTHONNET_WIN_DEFINE_CONSTANTS and PYTHONNET_MONO_DEFINE_CONSTANTS in…
Sep 7, 2017
afc90d5
Merge branch 'master' into xplat
filmor Sep 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Setup.py --xplat option refactored. Travis-ci build matrix extended.
  • Loading branch information
dse committed Aug 18, 2017
commit 3f02c12fdf34af8f7226e75b8998a7634201e39f
157 changes: 142 additions & 15 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,145 @@
dist: trusty
sudo: false

language: python
python:
- 2.7
- 3.3
- 3.4
- 3.5
- 3.6
- 3.7-dev


matrix:
allow_failures:
- python: 3.7-dev
include:
- python: 2.7
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe
- python: 3.3
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe
- python: 3.4
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe
- python: 3.5
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe
- python: 3.6
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe
- python: "3.7-dev"
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe

- python: 2.7
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
addons:
apt:
sources:
- sourceline: deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-trusty-prod trusty main
key_url: https://packages.microsoft.com/keys/microsoft.asc
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
- dotnet-hostfxr-2.0.0
- dotnet-runtime-2.0.0
- dotnet-sdk-2.0.0
- python: 3.3
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
addons:
apt:
sources:
- sourceline: deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-trusty-prod trusty main
key_url: https://packages.microsoft.com/keys/microsoft.asc
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
- dotnet-hostfxr-2.0.0
- dotnet-runtime-2.0.0
- dotnet-sdk-2.0.0
- python: 3.4
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
addons:
apt:
sources:
- sourceline: deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-trusty-prod trusty main
key_url: https://packages.microsoft.com/keys/microsoft.asc
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
- dotnet-hostfxr-2.0.0
- dotnet-runtime-2.0.0
- dotnet-sdk-2.0.0
- python: 3.5
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
addons:
apt:
sources:
- sourceline: deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-trusty-prod trusty main
key_url: https://packages.microsoft.com/keys/microsoft.asc
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
- dotnet-hostfxr-2.0.0
- dotnet-runtime-2.0.0
- dotnet-sdk-2.0.0
- python: 3.6
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
addons:
apt:
sources:
- sourceline: deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-trusty-prod trusty main
key_url: https://packages.microsoft.com/keys/microsoft.asc
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
- dotnet-hostfxr-2.0.0
- dotnet-runtime-2.0.0
- dotnet-sdk-2.0.0
- python: "3.7-dev"
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
addons:
apt:
sources:
- sourceline: deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-trusty-prod trusty main
key_url: https://packages.microsoft.com/keys/microsoft.asc
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
- dotnet-hostfxr-2.0.0
- dotnet-runtime-2.0.0
- dotnet-sdk-2.0.0
allow_failures:
- python: "3.7-dev"
env:
- BUILD_OPTS=
- NUNIT_PATH=./packages/NUnit.*/tools/nunit3-console.exe
- python: "3.7-dev"
env:
- BUILD_OPTS=--xplat
- NUNIT_PATH=~/.nuget/packages/nunit.consolerunner/3.*/tools/nunit3-console.exe
env:
global:
- LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so
Expand All @@ -23,8 +150,8 @@ env:
addons:
apt:
sources:
- mono
- mono-libtiff-compat
- sourceline: deb http://download.mono-project.com/repo/ubuntu trusty main
key_url: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xA6A19B38D3D831EF
packages:
- mono-devel
- ca-certificates-mono
Expand All @@ -37,11 +164,11 @@ before_install:
install:
- pip install --upgrade setuptools # TEMP - due to setuptools 36.2.0 bug
- pip install --upgrade -r requirements.txt
- coverage run setup.py install
- coverage run setup.py install $BUILD_OPTS

script:
- python -m pytest
- mono ./packages/NUnit.*/tools/nunit3-console.exe src/embed_tests/bin/Python.EmbeddingTest.dll
- mono $NUNIT_PATH src/embed_tests/bin/Python.EmbeddingTest.dll

after_script:
# Uncomment if need to geninterop, ie. py37 final
Expand Down
92 changes: 64 additions & 28 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import sys
import sysconfig
from distutils import spawn
from distutils.command import build_ext, install_data, install_lib
from distutils.command import install, build, build_ext, install_data, install_lib

from setuptools import Extension, setup

Expand Down Expand Up @@ -131,6 +131,12 @@ def _get_long_description():
except ImportError:
return '.Net and Mono integration for Python'

def _update_xlat_devtools():
global DEVTOOLS
if DEVTOOLS == "MsDev":
DEVTOOLS = "MsDev15"
elif DEVTOOLS == "Mono":
DEVTOOLS = "dotnet"

class BuildExtPythonnet(build_ext.build_ext):
user_options = build_ext.build_ext.user_options + [
Expand All @@ -144,6 +150,9 @@ def finalize_options(self):
build_ext.build_ext.finalize_options(self)

def build_extension(self, ext):
if self.xplat:
_update_xlat_devtools()

"""Builds the .pyd file using msbuild or xbuild"""
if ext.name != "clr":
return build_ext.build_ext.build_extension(self, ext)
Expand Down Expand Up @@ -174,7 +183,7 @@ def build_extension(self, ext):
if CONFIG == "Debug":
defines.extend(["DEBUG", "TRACE"])

if sys.platform != "win32" and DEVTOOLS == "Mono":
if sys.platform != "win32" and (DEVTOOLS == "Mono" or DEVTOOLS == "dotnet"):
on_darwin = sys.platform == "darwin"
defines.append("MONO_OSX" if on_darwin else "MONO_LINUX")

Expand Down Expand Up @@ -206,20 +215,34 @@ def build_extension(self, ext):
if DEVTOOLS == "MsDev":
_xbuild = '"{0}"'.format(self._find_msbuild_tool("msbuild.exe"))
_config = "{0}Win".format(CONFIG)

_solution_file = 'pythonnet.sln'
_custom_define_constants = False
elif DEVTOOLS == "MsDev15":
# Improve this with self._find_msbuild_tool_15 to find good >15.3 msbuild, currently only works under VS 15.3 developer environment.
_xbuild = '"{0}"'.format(self._find_msbuild_tool("msbuild.exe"))
_config = "{0}Win".format(CONFIG)
_solution_file = 'pythonnet.15.sln'
_custom_define_constants = True
elif DEVTOOLS == "Mono":
_xbuild = 'dotnet msbuild' if self.xplat else 'xbuild'
_xbuild = 'xbuild'
_config = "{0}Mono".format(CONFIG)
_solution_file = 'pythonnet.sln'
_custom_define_constants = False
elif DEVTOOLS == "dotnet":
_xbuild = 'dotnet msbuild'
_config = "{0}Mono".format(CONFIG)
Copy link
Copy Markdown
Contributor

@den-run-ai den-run-ai Sep 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmitriyse why dotnet DEVTOOLS corresponds to Mono _config?

Copy link
Copy Markdown
Contributor Author

@dmitriyse dmitriyse Sep 21, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first problem:
DebugMonoPY3, ReleaseMono, etc. build configuration is a historically names from times where nothing than mono had exists for the linux.
Now it is more correctly to name DebugLinuxPY3, ReleaseLinux, etc.

But we "cannot just rename" build configurations without affecting users.
We should introduce new build configurations (DebugLinuxPY3, ReleaseLinux), give a time to users to migrate to new names, and than remove old names DebugMonoPY3, ReleaseMono, etc.

The second problem:
We cannot use dotnet build command under windows due to DllExport project msbuild targets. We needs to use msbuild >=15 under windows

The result:

DEVTOOLS=dotnet ==> linux (--xplat) === ReleaseLinuxPY3 <== ReleaseMonoPY3

So currently we have DOVTOOLS=dotnet <===> ReleaseMonoPY3

If I am wrong in somewhere in my conclusions, lets improve something.

_solution_file = 'pythonnet.15.sln'
_custom_define_constants = True
else:
raise NotImplementedError(
"DevTool {0} not supported (use MsDev/Mono)".format(DEVTOOLS))
"DevTool {0} not supported (use MsDev/MsDev15/Mono/dotnet)".format(DEVTOOLS))

cmd = [
_xbuild,
'pythonnet.15.sln' if self.xplat else 'pythonnet.sln',
_solution_file,
'/p:Configuration={}'.format(_config),
'/p:Platform={}'.format(ARCH),
'/p:{}DefineConstants="{}"'.format('Custom' if self.xplat else '','%3B'.join(defines)),
'/p:{}DefineConstants="{}"'.format('Custom' if _custom_define_constants else '','%3B'.join(defines)),
'/p:PythonBuildDir="{}"'.format(os.path.abspath(dest_dir)),
'/p:PythonInteropFile="{}"'.format(os.path.basename(interop_file)),
'/verbosity:{}'.format(VERBOSITY),
Expand All @@ -230,16 +253,16 @@ def build_extension(self, ext):
cmd.append('/p:PythonManifest="{0}"'.format(manifest))

self.debug_print("Building: {0}".format(" ".join(cmd)))
use_shell = True if DEVTOOLS == "Mono" else False
use_shell = True if DEVTOOLS == "Mono" or DEVTOOLS == "dotnet" else False

subprocess.check_call(" ".join(cmd + ["/t:Clean"]), shell=use_shell)
subprocess.check_call(" ".join(cmd + ["/t:Build"]), shell=use_shell)

if DEVTOOLS == "Mono":
if DEVTOOLS == "Mono" or DEVTOOLS == "dotnet":
self._build_monoclr()

def _get_manifest(self, build_dir):
if DEVTOOLS != "MsDev":
if DEVTOOLS != "MsDev" and DEVTOOLS != "MsDev15":
return
mt = self._find_msbuild_tool("mt.exe", use_windows_sdk=True)
manifest = os.path.abspath(os.path.join(build_dir, "app.manifest"))
Expand Down Expand Up @@ -272,33 +295,30 @@ def _build_monoclr(self):

def _install_packages(self):
"""install packages using nuget"""
nuget = os.path.join("tools", "nuget", "nuget.exe")
use_shell = False
if DEVTOOLS == "Mono":
nuget = "mono {0}".format(nuget)
use_shell = True
use_shell = DEVTOOLS == "Mono" or DEVTOOLS == "dotnet"

if self.xplat:
if DEVTOOLS == "MsDev":
if DEVTOOLS == "MsDev15" or DEVTOOLS == "dotnet":
if DEVTOOLS == "MsDev15":
_config = "{0}Win".format(CONFIG)
elif DEVTOOLS == "Mono":
elif DEVTOOLS == "dotnet":
_config = "{0}Mono".format(CONFIG)
else:
raise NotImplementedError(
"DevTool {0} not supported (use MsDev/Mono)".format(DEVTOOLS))

cmd = "dotnet msbuild /t:Restore pythonnet.15.sln /p:Configuration={0} /p:Platform={1}".format(_config, ARCH)
self.debug_print("Updating packages with xplat: {0}".format(cmd))
subprocess.check_call(cmd, shell=use_shell)
return;
else:
nuget = os.path.join("tools", "nuget", "nuget.exe")

cmd = "{0} update -self".format(nuget)
self.debug_print("Updating NuGet: {0}".format(cmd))
subprocess.check_call(cmd, shell=use_shell)
if DEVTOOLS == "Mono":
nuget = "mono {0}".format(nuget)

cmd = "{0} restore pythonnet.sln -o packages".format(nuget)
self.debug_print("Installing packages: {0}".format(cmd))
subprocess.check_call(cmd, shell=use_shell)
cmd = "{0} update -self".format(nuget)
self.debug_print("Updating NuGet: {0}".format(cmd))
subprocess.check_call(cmd, shell=use_shell)

cmd = "{0} restore pythonnet.sln -o packages".format(nuget)
self.debug_print("Installing packages: {0}".format(cmd))
subprocess.check_call(cmd, shell=use_shell)

def _find_msbuild_tool(self, tool="msbuild.exe", use_windows_sdk=False):
"""Return full path to one of the Microsoft build tools"""
Expand Down Expand Up @@ -381,6 +401,21 @@ def run(self):

return install_data.install_data.run(self)

class InstallPythonnet(install.install):
user_options = install.install.user_options + [
('xplat', None, None)
]
def initialize_options(self):
install.install.initialize_options(self)
self.xplat = None

def finalize_options(self):
install.install.finalize_options(self)

def run(self):
if self.xplat:
_update_xlat_devtools()
return install.install.run(self)

###############################################################################
setupdir = os.path.dirname(__file__)
Expand Down Expand Up @@ -410,6 +445,7 @@ def run(self):
]),
],
cmdclass={
"install": InstallPythonnet,
"build_ext": BuildExtPythonnet,
"install_lib": InstallLibPythonnet,
"install_data": InstallDataPythonnet,
Expand Down