Bug/825 setitem slice dndarrays

[coquelin77]

Description

Fix for setting DNDarray values with a DNDarray which has a different size in the split dimension, could use cleaning.

Issue/s resolved: #825

Changes proposed:

• Fix for setting DNDarray values with a DNDarray which has a different size in the split dimension
• added OOP redistribute function
• add lshape_map property (not used everywhere yet)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Due Diligence

• All split configurations tested
• Multiple dtypes tested in relevant functions
• Documentation updated (if needed)
• Updated changelog.md under the title "Pending Additions"

Does this change modify the behaviour of other functions? If so, which?

no

[codecov]

Codecov Report

Merging #826 (89fb977) into master (b2939e5) will increase coverage by 0.02%.
The diff coverage is 98.44%.

@@            Coverage Diff             @@
##           master     #826      +/-   ##
==========================================
+ Coverage   95.30%   95.32%   +0.02%     
==========================================
  Files          64       64              
  Lines        9004     9056      +52     
==========================================
+ Hits         8581     8633      +52     
  Misses        423      423              

Flag	Coverage Δ
gpu	94.46% <98.44%> (+0.03%)	⬆️
unit	90.94% <98.44%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
heat/core/dndarray.py	97.30% <98.37%> (+0.18%)	⬆️
heat/core/manipulations.py	99.08% <100.00%> (+<0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2939e5...89fb977. Read the comment docs.

[mtar]

Keep it up! @coquelin77

I played around a little bit. It doesn't work when the split axes are different. Do you want to handle them too?

Some notes I took when reviewing the code. You wanted to clean it up anyway.

[ClaudiaComito]

I love the new lshape_map property. Brilliant.

A problem I'm having with setitem is this:

>>> import heat as ht
>>> a = ht.arange(10).reshape(2,5)
>>> a[1,ht.array([1])]
DNDarray([6], dtype=ht.int32, device=cpu:0, split=None)
>>> a[1,ht.array(1)]
DNDarray(6, dtype=ht.int32, device=cpu:0, split=None)
a[1,ht.array([1])] = 7
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/c.comito/HAF/heat/heat/core/dndarray.py", line 1404, in __setitem__
    return self.__setter(key, value)  # returns None
  File "/Users/c.comito/HAF/heat/heat/core/dndarray.py", line 1570, in __setter
    self.__array.__setitem__(key, value)
  File "/Users/c.comito/HAF/heat/heat/core/dndarray.py", line 976, in __len__
    return self.shape[0] 

This is not specific to this branch but maybe can be addressed here.

Thanks a lot!

[coquelin77]

Keep it up! @coquelin77

I played around a little bit. It doesn't work when the split axes are different. Do you want to handle them too?

Some notes I took when reviewing the code. You wanted to clean it up anyway.

do you mean to the value or the key for the different split axes?

if you mean the value is the one with a different split, then that is complicated, if the shapes are the right size, i dont have a problem with it. but that is easier said than done

[mtar]

From the example in #825:

x = ht.ones((10, 10), split=0)
setting = ht.zeros((8, 8), split=1)
x[1:-1, 1:-1] = setting
print(x)

This doesn't work.

This works however

x = ht.ones((10, 10), split=0)
setting = ht.zeros((4, 8), split=1)
x[0:4, 1:-1] = setting
print(x)

[1,0]<stdout>:DNDarray([[1., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[1,0]<stdout>:          [1., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[1,0]<stdout>:          [1., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[1,0]<stdout>:          [1., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[1,0]<stdout>:          [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1,0]<stdout>:          [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1,0]<stdout>:          [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1,0]<stdout>:          [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1,0]<stdout>:          [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1,0]<stdout>:          [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], dtype=ht.float32, device=cpu:0, split=0)
[1,1]<stdout>:

[coquelin77]

with some digging. the example that works in this case, only works because of an accident where all of the values are gathered onto rank 0.

I think that setting a split DNDarray with another differently split DNDarray should be handled in another issue

[coquelin77]

rerun tests

[mtar]

Can you increase the coverage?

---

array indexing is not available a[ht.array([0,1])] = ht.ones(2), but this is another issue.

[mtar]

Pending Additions

[mtar]

😘

[mtar]

Do you have a test for the warning? No test according to codecov

[mtar]

no test according to codecov