Numpy116 migration

[hmaarrfk]

@conda-forge/arm-arch I don't have right on the bot's branch. so here goes.
Checklist

• Used a fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[hmaarrfk]

@conda-forge/arm-arch do any of you have time to look at these failures? Feel free to push to my branch, these are kinda beyond my skill level.

[hmaarrfk]

@conda-forge-admin please rerender

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-webservice.

I tried to re-render for you, but it looks like there was nothing to do.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[hmaarrfk]

Executable file is not ELF:

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=26891&view=logs&jobId=544de720-5bd1-54e2-b0a2-e430dc2dba91&taskId=8befd53e-126a-5450-1c8e-4ab9ea027269&lineStart=3577&lineEnd=3589&colStart=1&colEnd=69

[hmaarrfk]

@tylerjereddy might you know what would have changed between numpy 1.15 and 1.16 that could cause that obscure error?

We can get numpy 1.15 to compile on our systems.

[mariusvniekerk]

This may require using shippable (or other native alternatives) for aarch64

[hmaarrfk]

did you have shippable rerendering working?

[isuruf]

Building this using ubuntu 18.04 would make the package built here incompatible with centos7 and all the arm64v8 builds using numpy will have to be compiled with ubuntu 18.04 or newer.

Any reason not to use the run_docker_build.sh script?

[mariusvniekerk]

mostly just testing for now, to see if the non-native build is the cause of the compilation issue. Appears that it isn't the case

[hmaarrfk]

The exact container here is beside the point. I'm just hacking here on some old build scripts I have.

Shippable needs things like sudo, and very particular containers. The original error is still there, so I've ruled out the hypothesis that it was due to qemu

    numpy/core/src/multiarray/descriptor.c:2053:9: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
             Py_RETURN_FALSE;
             ^~~~~~~~~~~~~~~
    In file included from numpy/core/include/numpy/npy_math.h:548:0,
                     from numpy/core/src/npymath/npy_math_common.h:9,
                     from numpy/core/src/npymath/npy_math_complex.c.src:34:
    numpy/core/src/npymath/npy_math_internal.h.src: In function 'npy_cacoshf':
    numpy/core/src/npymath/npy_math_internal.h.src:482:12: internal compiler error: Segmentation fault
         return @kind@@c@(x, y);
                ^~~~~~~~~~~~~~~
    Please submit a full bug report,
    with preprocessed source if appropriate.

[hmaarrfk]

@mariusvniekerk the build error is still there, even in shippable

    numpy/core/src/multiarray/descriptor.c:2053:9: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
             Py_RETURN_FALSE;
             ^~~~~~~~~~~~~~~
    In file included from numpy/core/include/numpy/npy_math.h:548:0,
                     from numpy/core/src/npymath/npy_math_common.h:9,
                     from numpy/core/src/npymath/npy_math_complex.c.src:34:
    numpy/core/src/npymath/npy_math_internal.h.src: In function 'npy_cacoshf':
    numpy/core/src/npymath/npy_math_internal.h.src:482:12: internal compiler error: Segmentation fault
         return @kind@@c@(x, y);
                ^~~~~~~~~~~~~~~
    Please submit a full bug report,
    with preprocessed source if appropriate.

[isuruf]

@hmaarrfk, try unsetting CFLAGS to see if some of the default CFLAGS is causing the ICE.

[hmaarrfk]

@isuruf do you think it is tree vectorize?

CFLAGS=-ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -I$PREFIX/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix

[hmaarrfk]

this binstar token is wrong too.

[isuruf]

Can you do this only for aarch64? And also try going back to azure.

[hmaarrfk]

mostly done. gotta see what MACHINE is for aarch64

[hmaarrfk]

@conda-forge-admin, please rerender

[hmaarrfk]

better, thanks! Why do you like to remove set -ex. it is so useful for sh noobs like me.

[isuruf]

set -e is the default when conda-build invokes the script. set -x doesn't do anything other than echoing the command

[hmaarrfk]

i like to echo ;). it teaches me when I need to add quotes.

[hmaarrfk]

I'll delete shippable stuff when the build passes.

[isuruf]

i like to echo ;). it teaches me when I need to add quotes.

Sure. Please add it for other branches as well. I'd like to keep the build script same across branches.

[hmaarrfk]

@isuruf seems like it could be an issue with char *

numpy/numpy#12862

I mean for now, you already reverted the set -x so I won't take the effort for something that works.

[isuruf]

@conda-forge-admin, rerender

[hmaarrfk]

what changed?

[isuruf]

what changed?

target_platform was used and needs a rerender for that variable to be registered as used. I'm not sure why conda-build's default value was not used

[isuruf]

-ftree-vectorize might still be an issue

[tylerjereddy]

@hmaarrfk Not sure what's going on here from a very quick look--is there something specific you want from the NumPy side of things? We have Shippable ARMv8 runs in our CI & I've even built OpenBLAS on ARMv8 to hopefully better match wheels on other platforms in the future.

Just scanning through--are there specific flags we should test for / constrain on, that are needed for ARM sometimes but not currently used in our CI?

[hmaarrfk]

Can somebody with knowledge of PPC64LE comment on whether or not this is a real error, or a bug with the tests?

___________ TestComplexFunctions.test_loss_of_precision[complex256] ____________

self = <numpy.core.tests.test_umath.TestComplexFunctions object at 0x3fff78526310>
dtype = <type 'numpy.complex256'>

    @pytest.mark.parametrize('dtype', [np.complex64, np.complex_, np.longcomplex])
    def test_loss_of_precision(self, dtype):
        """Check loss of precision in complex arc* functions"""
    
        # Check against known-good functions
    
        info = np.finfo(dtype)
        real_dtype = dtype(0.).real.dtype
        eps = info.eps
    
        def check(x, rtol):
            x = x.astype(real_dtype)
    
            z = x.astype(dtype)
            d = np.absolute(np.arcsinh(x)/np.arcsinh(z).real - 1)
            assert_(np.all(d < rtol), (np.argmax(d), x[np.argmax(d)], d.max(),
                                      'arcsinh'))
    
            z = (1j*x).astype(dtype)
            d = np.absolute(np.arcsinh(x)/np.arcsin(z).imag - 1)
            assert_(np.all(d < rtol), (np.argmax(d), x[np.argmax(d)], d.max(),
                                      'arcsin'))
    
            z = x.astype(dtype)
            d = np.absolute(np.arctanh(x)/np.arctanh(z).real - 1)
            assert_(np.all(d < rtol), (np.argmax(d), x[np.argmax(d)], d.max(),
                                      'arctanh'))
    
            z = (1j*x).astype(dtype)
            d = np.absolute(np.arctanh(x)/np.arctan(z).imag - 1)
            assert_(np.all(d < rtol), (np.argmax(d), x[np.argmax(d)], d.max(),
                                      'arctan'))
    
        # The switchover was chosen as 1e-3; hence there can be up to
        # ~eps/1e-3 of relative cancellation error before it
    
        x_series = np.logspace(-20, -3.001, 200)
        x_basic = np.logspace(-2.999, 0, 10, endpoint=False)
    
        if dtype is np.longcomplex:
            # It's not guaranteed that the system-provided arc functions
            # are accurate down to a few epsilons. (Eg. on Linux 64-bit)
            # So, give more leeway for long complex tests here:
            # Can use 2.1 for > Ubuntu LTS Trusty (2014), glibc = 2.19.
>           check(x_series, 50.0*eps)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python2.7/site-packages/numpy/core/tests/test_umath.py:2596: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = array([1.00000000e-20, 1.21736864e-20, 1.48198641e-20, 1.80412378e-20,
       ...871e-04, 6.73218092e-04, 8.19554595e-04, 9.97700064e-04],
      dtype=float128)
rtol = 1.23259516440783094595582588325435e-30

    def check(x, rtol):
        x = x.astype(real_dtype)
    
        z = x.astype(dtype)
        d = np.absolute(np.arcsinh(x)/np.arcsinh(z).real - 1)
        assert_(np.all(d < rtol), (np.argmax(d), x[np.argmax(d)], d.max(),
>                                 'arcsinh'))
E       AssertionError: (131, 1.54987431981169249551435343964035e-09, 4.0035173453529620614007210953362e-19, 'arcsinh')

[hmaarrfk]

Well if somebody can confirm that the ppc64le are OK failures, I think I just included the patch in the recipe, and the selector would just have to be changed.

[tylerjereddy]

@hmaarrfk Are you using genuine ppc64le architecture? I don't see shippable in the CI matrix for this PR & we don't see the failure on NumPy master with ppc64le / shippable.

That said, I do see the exact failure you mention when I run the NumPy test suite on WSL (Windows Subsystem for Linux), so I wouldn't be surprised if some other kind of "emulation" experiences a similar failure.

I'm sure floating point calc experts know far more about this stuff..

[hmaarrfk]

@tylerjereddy I think marius got native PPC64LE running on travis
https://travis-ci.org/conda-forge/numpy-feedstock/builds/525758989

Any idea why windows is randomly failing?

[tylerjereddy]

oh, right I'm confusing ppc64le and ARM -- don't think we see that on master either though, since we do the Travis ppc64le testing too

[hmaarrfk]

yeah, thats why im half worried. ill try to see if any prs were made affecting that in master.

Do you mind backporting
numpy/numpy#12709
Just to see the results on 1.16.3

[hmaarrfk]

This seems to be the most recent backport that touches that part of the code.
numpy/numpy#8566
numpy/numpy#12378

[hmaarrfk]

@conda-forge-admin please rerender

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-webservice.

I tried to re-render for you but ran into some issues, please ping conda-forge/core for further assistance.

[hmaarrfk]

Is this OK???

self = <numpy.core.tests.test_einsum.TestEinsum object at 0x3fff8be3c8d0>

    def test_einsum_sums_cfloat64(self):
>       self.check_einsum_sums('c8')

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.6/site-packages/numpy/core/tests/test_einsum.py:561: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numpy.core.tests.test_einsum.TestEinsum object at 0x3fff8be3c8d0>
dtype = 'c8', do_opt = False

    def check_einsum_sums(self, dtype, do_opt=False):
        # Check various sums.  Does many sizes to exercise unrolled loops.
    
        # sum(a, axis=-1)
        for n in range(1, 17):
            a = np.arange(n, dtype=dtype)
            assert_equal(np.einsum("i->", a, optimize=do_opt),
                         np.sum(a, axis=-1).astype(dtype))
            assert_equal(np.einsum(a, [0], [], optimize=do_opt),
                         np.sum(a, axis=-1).astype(dtype))
    
        for n in range(1, 17):
            a = np.arange(2*3*n, dtype=dtype).reshape(2, 3, n)
            assert_equal(np.einsum("...i->...", a, optimize=do_opt),
                         np.sum(a, axis=-1).astype(dtype))
            assert_equal(np.einsum(a, [Ellipsis, 0], [Ellipsis], optimize=do_opt),
                         np.sum(a, axis=-1).astype(dtype))
    
        # sum(a, axis=0)
        for n in range(1, 17):
            a = np.arange(2*n, dtype=dtype).reshape(2, n)
            assert_equal(np.einsum("i...->...", a, optimize=do_opt),
                         np.sum(a, axis=0).astype(dtype))
            assert_equal(np.einsum(a, [0, Ellipsis], [Ellipsis], optimize=do_opt),
                         np.sum(a, axis=0).astype(dtype))
    
        for n in range(1, 17):
            a = np.arange(2*3*n, dtype=dtype).reshape(2, 3, n)
            assert_equal(np.einsum("i...->...", a, optimize=do_opt),
                         np.sum(a, axis=0).astype(dtype))
            assert_equal(np.einsum(a, [0, Ellipsis], [Ellipsis], optimize=do_opt),
                         np.sum(a, axis=0).astype(dtype))
    
        # trace(a)
        for n in range(1, 17):
            a = np.arange(n*n, dtype=dtype).reshape(n, n)
            assert_equal(np.einsum("ii", a, optimize=do_opt),
                         np.trace(a).astype(dtype))
            assert_equal(np.einsum(a, [0, 0], optimize=do_opt),
                         np.trace(a).astype(dtype))
    
        # multiply(a, b)
        assert_equal(np.einsum("..., ...", 3, 4), 12)  # scalar case
        for n in range(1, 17):
            a = np.arange(3 * n, dtype=dtype).reshape(3, n)
            b = np.arange(2 * 3 * n, dtype=dtype).reshape(2, 3, n)
            assert_equal(np.einsum("..., ...", a, b, optimize=do_opt),
                         np.multiply(a, b))
            assert_equal(np.einsum(a, [Ellipsis], b, [Ellipsis], optimize=do_opt),
                         np.multiply(a, b))
    
        # inner(a,b)
        for n in range(1, 17):
            a = np.arange(2 * 3 * n, dtype=dtype).reshape(2, 3, n)
            b = np.arange(n, dtype=dtype)
>           assert_equal(np.einsum("...i, ...i", a, b, optimize=do_opt), np.inner(a, b))
E           AssertionError: 
E           Arrays are not equal
E           
E           Mismatch: 50%
E           Max absolute difference: 28.
E           Max relative difference: 0.056
E            x: array([[ 204.+0.j,  528.+0.j,  852.+0.j],
E                  [1176.+0.j, 1500.+0.j, 1824.+0.j]], dtype=complex64)
E            y: array([[ 204.+0.j,  500.+0.j,  852.+0.j],
E                  [1148.+0.j, 1500.+0.j, 1796.+0.j]], dtype=complex64)

[hmaarrfk]

@conda-forge-admin, please rerender

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-webservice.

I tried to re-render for you, but it looks like there was nothing to do.

[hmaarrfk]

sigh the aarch build got stuck, but i'll save CI resources and just leave it at that.

[hmaarrfk]

@conda-forge/arm-arch does anybody want to give this a last review and a good push?

[hmaarrfk]

I guess we just update the pinning in recipes to be numpy 1.16? and call it a day?