Outstanding - Integrate DataManager with cube

[bjlittle]

Ref - #2492

The following outstanding issues still require to be investigated and/or addressed:

• fix cube arithmetic dtype, see Fix cube arithmetic dtype #2514
• fix cube concatenate and merge (see Dask cube and data manager integration #2492)
  • currently dask cube does not merge an ndarray and masked array to generate a masked result
  • we need to iterate through all permutations or real vs lazy, and ndarray vs masked to gain confidence
  • need to adopt the concept of a cube with an empty fill_value of None matching a candidate cube with an existing fill_value - although two candidate cubes with difference non-None fill_values should never match (obviously)
  • also relates to Ensure adequate testing for lazy data merge and concatenate (3 man days) #2384
• investigate and implement setting cube.fill_value through a cube.data = masked_array operation, see this comment for context, see Populate cube.fill_value thru cube.data setter. #2524
• Set cube.fill_value to None when merging or concatenating a mixture of masked array and ndarray, see Dask merge concat fill value #2520
• Consider whether to issue a warning in cube.data getter when setting the fill_value to None, as the returned masked array will contain a numpy default fill_value

[pp-mo]

need to adopt the concept of a cube with an empty fill_value of None matching a candidate cube with an existing fill_value - although two candidate cubes with difference non-None fill_values should never match (obviously)

I disagree with this.
I think, if you are including data with "no known suitable fill value", the result has ... "no known suitable fill value".
For instance,

>>> np.stack(
...     [ma.array([0, 1, 2], fill_value=-1),
...      ma.array([-1, -2, -3])])
masked_array(data =
 [[ 0  1  2]
 [-1 -2 -3]],
             mask =
 False,
       fill_value = 999999)

>>> 

That's equivalent to a merge. In this case, "-1" is not a suitable fill value for the result, as it has a valid -1 point in it.

[djkirkham]

Cube merging now (once #2492 is merged) allows cubes to merge if one of them has a fill value and the other doesn't. But this causes problems given a list where one of the cubes has no fill value and the others have differing fill values: the first cube matched with the None-fill-value cube will be merged with it, and subsequent cubes won't, making merging dependent on the order of the cubes in the list.

I'm not sure what the correct behaviour should be. If we want to keep fill value promotion, then it seems like we need logic along the lines of

For any cubes A, B and C in a list, if A can merge with each of B and C, then merging should fail unless B can merge with C.

In terms of fill values, that amounts to

For any cubes A, B and C in a list, if A has a fill value of None, then merging should fail unless B and C have the same fill value or one of them has a flll value of None.

The problem is that merging assumes the match method of _CubeSignature is transitive, which is no longer the case. With this assumption, the first cube in the list can be used as a reference for other cubes to match against without requiring comparisons with the remaining cubes.

Actually, the more I think about it the more it seems the best approach is to not allow fill_value promotion, at least not at this point. I think allowing it requires a significant amount of work which we don't have time for. But that's just my opinion.

[djkirkham]

Cubes with lazy integer masked data will keep their integer dtype when operations are performed on them which should result in a change of dtype (e.g. log). For example:

from numpy import ma
from iris.cube import Cube
from iris.analysis.maths import log
from iris._lazy_data import as_lazy_data
a = ma.masked_array([10,20], [0,1])
la = as_lazy_data(a)
cube = Cube(la, dtype=int)
cube = log(cube)
print cube.dtype
print cube.data

yields:

int64
[2 --]

[bjlittle]

@djkirkham Can you create a separate issue for this or are you already addressing this somewhere?

[djkirkham]

@bjlittle I don't have permissions to create issues within any of the dask projects. I'm not currently working on this, but what I've added in #2526 could be used as part of the solution I think.

[bjlittle]

@djkirkham Let me check the perms ... or at least understand why.

[bjlittle]

@djkirkham But you can at least create an issue on iris, right?

And set the label and milestone to dask? ... if not, just quote the issue number as a comment here and I'll add it to the board.

I can't see how to set permissions for a user specifically for a project, which is a shame 😖

[djkirkham]

@bjlittle I can't set the milestone, but here's the issue: #2528