Use miniconda for dependencies on Travis - #3273

https://github.com/soft-matter/trackpy/blob/master/.travis.yml

https://github.com/soft-matter/pims/blob/master/.travis.yml

Are the two projects I work on that I know use mini-conda. These are (mostly) the work of @danielballan

I think this is to lower the time spend on installing numpy.
Could we use wheels? That should in theory help.

One can use wheels, and astropy used to do this, but then the burden is on someone to produce the wheels and keep them up to date, since numpy itself does not provide wheels for Ubuntu, only Mac.

The advantage of miniconda is that multiple versions of numpy are always available for multiple versions of python.

Another advantage of going with miniconda is can manage the c-dependencies for us as well. This means we can pick a version of freetype, pin it, and then make our image comparisons very stringent.

I thought numpy provided wheels for every distro. Too bad…

Providing wheels for linux-based distros is basically useless (at least,
back when wheels started being supported by PyPi):
https://bitbucket.org/pypa/pypi/issue/120/binary-wheels-for-linux-are-not-supported
https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for

I would love to see any update on this front, though.

On Fri, Jul 18, 2014 at 1:32 PM, Varoquaux notifications@github.com wrote:

I thought numpy provided wheels for every distro. Too bad…

—
Reply to this email directly or view it on GitHub
#3273 (comment)
.

@WeatherGod: That limitation of wheels is unfortunate, but a really hard problem, now that I think about it. The advantage miniconda has is that they abstract away enough of the underlying platform, such that the package is for "conda", not "conda-on-Ubuntu" or "conda-on-Fedora". IIRC, I think Aaron even mentioned that quickly in his Scipy talk.

Limitation of wheels can be solved by using custom http directory for
travis wheels. nipy and astropy do this. Example:
https://nipy.bic.berkeley.edu/scipy_installers/travis/

It's fairly easy to build wheels on the travis virtual machine or something
like it using Vagrant :
https://gist.github.com/matthew-brett/714b50bd3159d416981a

On Fri, Jul 18, 2014 at 2:07 PM, Michael Droettboom <
notifications@github.com> wrote:

@WeatherGod https://github.com/WeatherGod: That limitation of wheels is
unfortunate, but a really hard problem, now that I think about it. The
advantage miniconda has is that they abstract away enough of the underlying
platform, such that the package is for "conda", not "conda-on-Ubuntu" or
"conda-on-Fedora". IIRC, I think Aaron even mentioned that quickly in his
Scipy talk.

—
Reply to this email directly or view it on GitHub
#3273 (comment)
.

@matthew-brett -- Astropy and affiliated packages moved away from wheels to conda because it turned out to be a headache to use wheels: the biggest challenge was that some dependencies (like scipy) need to be built against the same specific numpy version as anything else that depended on numpy (e.g. astropy).

The wheel naming scheme doesn't allow for specifying both a python version number and a numpy version number. We tried working around that for a while (see astropy/package-template#44 for some of the details, or contact me).

You can see the scripts we wrote to try to get the wheel building working at astropy/astropy-wheels#1

That's odd - I thought that the numpies just had to be binary compatible? I mean, if you build against numpy 1.5.1 then you can use the wheels with numpy >= 1.5.1.

But in any case - can't you just build the wheels on the something like the vagrant travis machine and not worry about that? https://gist.github.com/matthew-brett/714b50bd3159d416981a

Sorry - have had few glasses of wine so might not be thinking straight.

You are right that in principle the numpy releases are ABI compatible...but in practice they are not (and there were some API changes from 1.5 to 1.6 (or 1.6 to 1.7) in addition). The specific issue that came up is described in astropy/package-template#37

The building wasn't too much of a problem; a vagrant solution like your would work. I built on a local VM because it needed to be done infrequently, and I've never used vagrant :).

IIRC, two things pushed us over the line from wheels to conda:

• complexity -- when we added matplotlib as one of the wheels to build we needed to worry both about numpy version and scipy vresion. Using conda packages solved most of that.
• speed -- conda packages download on travis at insanely fast speeds, tens of MB/sec vs a few MB/sec from either of the wheelhouses astropy had set up.

To be clear, I don't have any opinion at all about what matplotlib uses, just thought it would be useful to share the astropy experience.

Ah - I see what you mean - I ran into this too. The issue is that numpies are forwards but not backwards compatible. That is, you can run against a later numpy version than the one you built against, but not an earlier version.

I don't know how conda installs against a particular numpy package, but I assume they do the same thing that I do when building scipy OSX wheels, which is to build packages against the earliest compatible numpy; in that case the scipy package is the same regardless of the numpy installed.

I guess you hit this problem because you have an explicit grid testing against multiple numpy versions per Python version.

I can't give much detail as to speed, but the scikit-image setup I just made using wheels seems to be running in the same ballpark as the conda setup.

I guess you hit this problem because you have an explicit grid testing against multiple numpy versions per Python version.

Exactly; conda is set up so that conda install python=X.Y numpy=A.B scipy only installs a scipy that matches python vX.Y and numpy vA.B.

Right - but I guess that conda doesn't do anything fancy to make the scipy package match, it only makes sure that the numpy they build scipy against is the minimum version that is compatible with scipy. It's the same deal for wheels, you just have to know about the forward-backward compatibility thing when building the wheels.

So I'm guessing that whether you do conda install python=2.7 numpy=1.5 scipy or conda install python=2.7 numpy=1.8 scipy you get the same scipy package in either case.

So... this is now working. But it doesn't make compile times any faster. It seems the differences are within the noise of Travis runtimes. Given that, is there much reason to do this?

Conda would make it easier to test the gui backends. There are some tests not currently running since PyQt is missing but I am not sure if it is worth the install time for installing PyQt

@mdboom This needs a re-base

@mdboom This was auto-closed when I removed the 1.4.x branch. Can you rebase and re-target this at master?

nm, I recreated the PR my self.