Anaconda org weekly upload

[tylerjereddy]

Supersedes #68 because testing the secret API key for anaconda.org uploads likely requires that the feature branch originates in the repo itself.

Don't merge yet---still in a debug state with matrix largely commented out.

[tylerjereddy]

pypi-format file upload has been observed from this PR: https://anaconda.org/scipy-wheels-nightly/scipy/files

I've therefore restored the full wheels build matrices & also restored the requirement that the upload only happens for the weekly cron jobs, for now.

[ogrisel]

Note that the pip compatible URL is:

https://pypi.anaconda.org/scipy-wheels-nightly/simple/scipy

or:

https://pypi.anaconda.org/scipy-wheels-nightly/simple

to get all the nightly build wheels at once.

[ogrisel]

There is something I do not understand: where is defined the version number openblas embedded in the scipy wheel?

We would need to download the openblas-libs from the new anaconda staging URL but I cannot find it in the travis configuration used for the Linux and macOS builds.

Is the manylinux wheel using the libopenblas that comes with the docker image?

For macOS, I think OpenBLAS is fetched using the get_gf_lib functions from the rackspace URL as defined in:

multibuild/library_builders.sh:GF_LIB_URL="https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com"

and the OPENBLAS_VERSION env variables seems to be left to the default version:

multibuild/library_builders.sh:OPENBLAS_VERSION="${OPENBLAS_VERSION:-0.2.18}"

which means that more recent builds from https://github.com/MacPython/openblas-libs are not used for macOS and Linux. Am I correct?

[mattip]

It comes from env_vars.sh which defines $OPENBLAS_VERSION which is used by config.sh in a call to build_libs

[mattip]

In the NumPy CI builds, we have moved to a python-based openblas_support.py which also provides the ability to test that the correct openblas library is actually used. I am testing out moving numpy-wheels to use this instead in MacPython/numpy-wheels#73

[tylerjereddy]

Ah right, I should probably try to disable the use of rackspace URLs for the OpenBLAS downloads here. Forgot about that.

@mattip Is it obvious to you what I would use in SciPy to substitute for:

dll = ctypes.CDLL(numpy.core._multiarray_umath.__file__)

@ogrisel I noticed that you were using pip to install anaconda-client in another PR. Do you think that might be more efficient than what I do here? It does seem a little awkward for me to set up a whole conda env just to use anaconda-client?

[mattip]

dll = ctypes.CDLL(numpy.core._multiarray_umath.file)

Any of the files that use openblas:

for f in `find build/lib.linux-x86_64-3.7/scipy/ -name '*.so'`; do echo $f; ldd $f |grep blas; done

This one seems like a likely candidate: linalg/_fblas.cpython-37m-x86_64-linux-gnu.so

[mattip]

or linalg/cython_blas.cpython-37m-x86_64-linux-gnu.so

[ogrisel]

@ogrisel I noticed that you were using pip to install anaconda-client in another PR. Do you think that might be more efficient than what I do here? It does seem a little awkward for me to set up a whole conda env just to use anaconda-client?

Yes installing from pip + git might be more efficient. At least it's fewer lines of scripting thant wget'ing the miniconda installer.

I opened anaconda/anaconda-client#540 upstream.

[tylerjereddy]

Ok, I've updated this now to use pip for a lighter-weight solution. I'm starting to get confused about all the different things going on, but I think maybe I'll try to just limit this PR to "getting weekly uploads to anaconda.org working."

[tylerjereddy]

@pv @rgommers CI looks ok here now (Travis 32-bit fails are from SciPy itself), so probably ready for review.

I think this is just for reactivating nightly wheels, with a bit more follow-up work for turning on the other wheel uploads (each time jobs succeed).

[rgommers]

Thanks Tyler! Let's give this a go:)