SPARK-1428: MLlib should convert non-float64 NumPy arrays to float64 instead of complaining

[techaddict]

No description provided.

[mengxr]

@techaddict Is it easy to add a test to verify that it works?

[techaddict]

@mengxr ok i'll try adding some test's

[mateiz]

Jenkins, this is ok to test

[mateiz]

You should also check that the vector does not contain complex numbers, since that is the one NumPy data type we can't convert to floats. You can do

if numpy.issubdtype(v.dtype, numpy.complex):
    raise TypeError(...)

[techaddict]

@mateiz updated, should i add tests too (IMHO i don't think there is need, because its a trivial patch ) ?

[mateiz]

Actually one more thing, can you set copy=true in astype (or just not set it)? I'm not sure that it preserves the right byte order and such and I know that this works with copy=true. Also we don't want to mess with the user's input data.

[mateiz]

Sorry meant true, not false.

[techaddict]

@mateiz You mean
v = v.astype(float64)
or v = v.astype(float64, copy=True)

[techaddict]

@mateiz is there any other problem ?

[mateiz]

This seems to be failing tests unfortunately -- click through to the Jenkins log. This is what it says:

=========================================================================
Running PySpark tests
=========================================================================
**********************************************************************
File "pyspark/mllib/_common.py", line 76, in __main__._deserialize_double_vector
Failed example:
    array_equal(x, _deserialize_double_vector(_serialize_double_vector(x)))
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest __main__._deserialize_double_vector[1]>", line 1, in <module>
        array_equal(x, _deserialize_double_vector(_serialize_double_vector(x)))
      File "pyspark/mllib/_common.py", line 55, in _serialize_double_vector
        if numpy.issubdtype(v.dtype, numpy.complex):
    NameError: global name 'numpy' is not defined
**********************************************************************
   1 of   2 in __main__._deserialize_double_vector
***Test Failed*** 1 failures.
Had test failures; see logs.

[mateiz]

This is why it would be good to add a test actually. For instance you can add a test with _deserialize_double_vector(_serialize_double_vector(array([1,2,3])) and check that it returns array([1.0,2.0,3.0]).

[techaddict]

@mateiz done working now 👍

[mateiz]

Thanks Sandeep. Merged into master and 1.0.

[AmplabJenkins]

Can one of the admins verify this patch?