python 3.7 issues

pycparser error:

https://travis-ci.org/pythonnet/pythonnet/jobs/334432678#L1991

pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW

I am seeing the same error when attempting to build Docker images for https://github.com/jonemo/pythonnet-docker for all combinations that involve Python 3.6.5:

Python 3.6.5, Mono 4.8.0.524, Python.NET 2.3.0
Python 3.6.5, Mono 4.8.0.524, Python.NET master branch
Python 3.6.5, Mono 5.2.0.224, Python.NET master branch
Python 3.6.5, Mono 5.4.1.6, Python.NET master branch

Python 3.5.5 and 3.4.8 are not affected.

Here is the output I get for the command pip install pythonnet==2.3.0 (almost identical to what's behind the Travis link above):

Step 3/3 : RUN pip install pycparser   && pip install pythonnet==2.3.0
 ---> Running in 306362b2258a
Collecting pycparser
  Downloading pycparser-2.18.tar.gz (245kB)
Building wheels for collected packages: pycparser
  Running setup.py bdist_wheel for pycparser: started
  Running setup.py bdist_wheel for pycparser: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/95/14/9a/5e7b9024459d2a6600aaa64e0ba485325aff7a9ac7489db1b6
Successfully built pycparser
Installing collected packages: pycparser
Successfully installed pycparser-2.18
Collecting pythonnet==2.3.0
  Downloading pythonnet-2.3.0.tar.gz (1.5MB)
Building wheels for collected packages: pythonnet
  Running setup.py bdist_wheel for pythonnet: started
  Running setup.py bdist_wheel for pythonnet: finished with status 'error'
  Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zrnh3win/pythonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp9rlc9nv2pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  Checking for updates from https://www.nuget.org/api/v2/.
  Currently running NuGet.exe 3.5.0.
  Updating NuGet.exe to 4.5.1.
  Update successful.
  MSBuild auto-detection: using msbuild version '14.0' from '/usr/lib/mono/xbuild/14.0/bin'.
  MSBUILD: error MSBUILD0004: Too many project files specified

  WARNING: Error reading msbuild project information, ensure that your input solution or project file is valid. NETCore and UAP projects will be skipped, only packages.config files will be restored.
  Restoring NuGet package NUnit.3.6.0.
  Restoring NuGet package NUnit.ConsoleRunner.3.6.0.
  Restoring NuGet package UnmanagedExports.1.2.7.
    GET https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.6.0/nunit.consolerunner.3.6.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit/3.6.0/nunit.3.6.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    OK https://api.nuget.org/v3-flatcontainer/nunit/3.6.0/nunit.3.6.0.nupkg 266ms
  Installing NUnit 3.6.0.
    OK https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg 910ms
  Installing UnmanagedExports 1.2.7.
    OK https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.6.0/nunit.consolerunner.3.6.0.nupkg 1216ms
  Installing NUnit.ConsoleRunner 3.6.0.
  Adding package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Adding package 'NUnit.ConsoleRunner.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Added package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Added package 'NUnit.ConsoleRunner.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Adding package 'NUnit.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'
  Added package 'NUnit.3.6.0' to folder '/tmp/pip-build-zrnh3win/pythonnet/packages'

  NuGet Config files used:
      /root/.config/NuGet/NuGet.Config

  Feeds used:
      https://api.nuget.org/v3/index.json

  Installed:
      3 package(s) to packages.config projects
  clang: warning: /usr/local/include/python3.6m: 'linker' input unused
  Traceback (most recent call last):
    File "tools/geninterop/geninterop.py", line 292, in <module>
      sys.exit(main())
    File "tools/geninterop/geninterop.py", line 274, in main
      ast = parser.parse(python_h)
    File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 152, in parse
      debug=debuglevel)
    File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 331, in parse
      return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
    File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
      tok = call_errorfunc(self.errorfunc, errtoken, self)
    File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
      r = errorfunc(token)
    File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 1761, in p_error
      column=self.clex.find_tok_column(p)))
    File "/usr/local/lib/python3.6/site-packages/pycparser/plyparser.py", line 66, in _parse_error
      raise ParseError("%s: %s" % (coord, msg))
  pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 405, in <module>
      zip_safe=False,
    File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 204, in run
      self.run_command('build')
    File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 191, in build_extension
      subprocess.check_call([sys.executable, geninterop, interop_file])
    File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'tools/geninterop/geninterop.py', 'src/runtime/interop36m.cs']' returned non-zero exit status 1.

  ----------------------------------------
  Failed building wheel for pythonnet
  Running setup.py clean for pythonnet
Failed to build pythonnet
Installing collected packages: pythonnet
  Running setup.py install for pythonnet: started
    Running setup.py install for pythonnet: finished with status 'error'
    Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zrnh3win/pythonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9h9w6f8b-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    Checking for updates from https://www.nuget.org/api/v2/.
    Currently running NuGet.exe 4.5.1.
    NuGet.exe is up to date.
    MSBuild auto-detection: using msbuild version '14.0' from '/usr/lib/mono/xbuild/14.0/bin'.
    MSBUILD: error MSBUILD0004: Too many project files specified

    WARNING: Error reading msbuild project information, ensure that your input solution or project file is valid. NETCore and UAP projects will be skipped, only packages.config files will be restored.
    All packages listed in packages.config are already installed.
    clang: warning: /usr/local/include/python3.6m: 'linker' input unused
    Traceback (most recent call last):
      File "tools/geninterop/geninterop.py", line 292, in <module>
        sys.exit(main())
      File "tools/geninterop/geninterop.py", line 274, in main
        ast = parser.parse(python_h)
      File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 152, in parse
        debug=debuglevel)
      File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 331, in parse
        return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
      File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
        tok = call_errorfunc(self.errorfunc, errtoken, self)
      File "/usr/local/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
        r = errorfunc(token)
      File "/usr/local/lib/python3.6/site-packages/pycparser/c_parser.py", line 1761, in p_error
        column=self.clex.find_tok_column(p)))
      File "/usr/local/lib/python3.6/site-packages/pycparser/plyparser.py", line 66, in _parse_error
        raise ParseError("%s: %s" % (coord, msg))
    pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 405, in <module>
        zip_safe=False,
      File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/usr/local/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/tmp/pip-build-zrnh3win/pythonnet/setup.py", line 191, in build_extension
        subprocess.check_call([sys.executable, geninterop, interop_file])
      File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/local/bin/python', 'tools/geninterop/geninterop.py', 'src/runtime/interop36m.cs']' returned non-zero exit status 1.

    ----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zrnh3win/pythonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9h9w6f8b-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-zrnh3win/pythonnet/
The command '/bin/sh -c pip install pycparser   && pip install pythonnet==2.3.0' returned a non-zero code: 1

Same issue here with python3.6.5 on a clean pipenv environment.

any hope of Windows pythonnet wheel under Python-3.7 ? where a few days before RC.

I have the same issue on the Windows Linux subsystem using (Python, Mono, Dotnet) = (3.6.5, 5.12, 2.1.300) building out of master. However, you can "cheat" to make it work until a fix is in place, as follows:

Create a conda env on a Python version that works (I used 3.6.2)
Build using python setup.py bdist_wheel --xplat
Install from you root (3.6.5) environment using pip install --no-index --find-links=.\dist\ pythonnet

(instruction from Installation page)

@fractus

@fractus @jonemo so which python 3.6.x version broke this pycparser parsing? then i can probably send this over to cpython bug tracker.

also if someone can attach debug files generated by pycparser/ply during this error, that would be helpful too.

It appears to be 3.6.5 -- it works on 3.6.{2, 3, 4}. On master, it fails on l. 220 of setup.py; to replicate from the commad line:

python tools/geninterop/geninterop.py src/runtime/interop36m.cs

Not sure how to generate the debug files of pycparser.

Ps. This is run in conda environments (conda create -n c36# python=3.6.#) plus pycparser 2.18.

@fractus

@fractus it appears that this additional header crypt.h was added in python 3.6.5 (backported) and we may need to either force exclude it in geniterop.py or add a fake clib include for it to pycparser. It is probably not c99 compliant header.

https://docs.python.org/3/whatsnew/changelog.html#id8
https://github.com/python/cpython/pull/5284/files
https://eli.thegreenplace.net/2015/on-parsing-c-type-declarations-and-fake-headers/

@denfromufa Not sure if this is what you're looking for, but I've attached a trace log file generated when trying to build pythonnet 2.3.0 from source on Pop!_OS 18.04 LTS (Linux x64). I obtained the source from https://anaconda.org/pythonnet/pythonnet/files . I get the same errors noted above.
trace.log.gz
Python -V = Python 3.6.5 :: Anaconda custom (64-bit)

@thomasmortiz

@thomasmortiz that trace does not contain the pycparser error. this is probably only tracing of python imports. actually we don't need any debug/log, just fix/avoid crypt.h parsing.

Python-3.7.0 is out.

@stonebig

@stonebig waiting for this 😸 appveyor/ci#2475

I added a py37 branch in the meantime with interop37.cs copied from interop36.cs as-is.

#698

appveyor seems ready, now.

the wheel pythonnet-2.4.0.dev0-cp37-cp37m-win32.whl from cgohlke works great on Python-3.7 !

This also occurs for me on Python 3.5.2

Well, you probably know better than me what this build chain does. But just be aware that Python 3.7 compiles fine in CPython's own Travis-CI setup ;-)

At least the command line looks unusual ('-I', 'tools/geninterop/fake_libc_include'?):
https://travis-ci.org/pythonnet/pythonnet/jobs/415741697#L2017

fake_libc_include from pycparser is documented pretty well and used all over to parse c-header files non-conformant to c99. And the build used to parse the headers for python37-dev.

…

On Tue, Aug 14, 2018, 12:56 PM Antoine Pitrou ***@***.***> wrote: Well, you probably know better than me what this build chain does. But just be aware that Python 3.7 compiles fine in CPython's own Travis-CI setup ;-) At least the command line looks unusual ('-I', 'tools/geninterop/fake_libc_include'?): https://travis-ci.org/pythonnet/pythonnet/jobs/415741697#L2017 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#609 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5awd_hsamDXQoIElUgk-ipsz0dBQks5uQw9FgaJpZM4Rvzar> .

ok, reported in pycparser: eliben/pycparser#276

Maybe we should try out this pycparser extended version or/and gcc parser:

https://github.com/inducer/pycparserext

#510

@eliben

This is now fixed, like @eliben suggested it was a missing #define to the compiler:

https://github.com/pythonnet/pythonnet/blob/py37/tools/geninterop/geninterop.py#L177

"-D _POSIX_THREADS"

Next is nuget/msbuild issue under Mono:

NuGet/Home#7202

Python 3.7 support is merged!

#698

oh, and this fixes reported parsing errors on Python 3.6+ with geninterop/pycparser/ply.

The pythonnet VS2017 GUI build fails for Python 3.7. Apparently due to not have interop37.cs in Python.Runtime.csproj (but I'm not a Visual Studio expert):

  <ItemGroup Condition=" '$(PythonInteropFile)' == '' ">
    <Compile Include="interop27.cs" />
    <Compile Include="interop34.cs" />
    <Compile Include="interop35.cs" />
    <Compile Include="interop36.cs" />
  </ItemGroup>

@Jeff17Robbins

@Jeff17Robbins my bad, can you please submit a PR to include this file to project file?

@denfromufa I'll get help from some actual programmers at work on this. I'm not too git savvy.

@Jeff17Robbins

some follow-up issues:

#720

@Jeff17Robbins you can just make this change from github UI without git.

adding iterop37.cs to Python.Runtime.csproj and removing old TODO com…

@paulie4

@denfromufa @paulie4 and I discussed, and he created the PR. As he already stated, we are only able to test with VS2017.

Also already stated, the UnmanagedExports tool doesn't seem to know how to find lib.exe in the VS2017 folder tree, and so it creates a pythonnet that doesn't actually work. Which can be really confusing!

The Release Notes on the nuget site says

no longer fails the build when it can't find lib.exe

But...that's not a good thing!

The failure shows up at runtime, not build time:

import clr

ImportError: dynamic module does not define init function (initclr)

We couldn't find the source code site for that UnmanagedExports tool, so we don't know how to fix it.

Navigation Menu

Repository navigation

python 3.7 issues #609

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

python 3.7 issues #609

Description

Activity

jonemo commented on Apr 2, 2018

qherreros commented on May 9, 2018

stonebig commented on May 27, 2018

fractus commented on Jun 2, 2018

den-run-ai commented on Jun 3, 2018

den-run-ai commented on Jun 3, 2018

fractus commented on Jun 3, 2018

den-run-ai commented on Jun 3, 2018

thomasmortiz commented on Jun 10, 2018

den-run-ai commented on Jun 12, 2018

stonebig commented on Jul 1, 2018

den-run-ai commented on Jul 5, 2018

stonebig commented on Jul 19, 2018

stonebig commented on Jul 22, 2018

CaerphillyMediaLtd commented on Jul 24, 2018

6 remaining items

pitrou commented on Aug 14, 2018

den-run-ai commented on Aug 14, 2018

den-run-ai commented on Aug 14, 2018

den-run-ai commented on Aug 15, 2018

den-run-ai commented on Aug 15, 2018

den-run-ai commented on Aug 15, 2018

den-run-ai commented on Oct 21, 2018

den-run-ai commented on Oct 21, 2018

Jeff17Robbins commented on Oct 27, 2018

den-run-ai commented on Oct 27, 2018

Jeff17Robbins commented on Oct 27, 2018

den-run-ai commented on Nov 1, 2018

Jeff17Robbins commented on Nov 1, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions