-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle filling aca image array differently #297
Conversation
mica/archive/aca_l0.py
Outdated
for i, row in enumerate(dat): | ||
imgraw = imgraws[i].reshape(8, 8) | ||
sz = imgsizes[i] | ||
if sz < 8: | ||
imgraw = imgraw[:sz, :sz] | ||
|
||
meta = {name: col[i] for name, col in cols.items() if col[i] != -9999} | ||
meta = {name: col[i] for name, col in cols.items() if col[i] != fill_vals[name]} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
imgraw.fill_value
is not -9999, but you filled it with -9999, so the condition if col[i] != fill_vals[name]
will never be true for IMGRAW
. Will that be a problem? Or IMGRAW is an exception?
It might be better to do
dat["IMGRAW"].fill_value = -9999
imgraws = dat["IMGRAW"].filled()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suppose the ambiguity is that I only made this work for the the default columns (IMGRAW isn't in the default columns) and it isn't expected that one would put the 8x8 imgraw data in the metadata for the image. I should probably just remove imgraw from columns if it is in there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And your idea to update the fill value for IMGRAW is also a good one. Technically I think I'd need to do that before making that little dictionary I have from column names to fill values to be 100% consistent (but the idea would still be that we don't have to worry about imgraw in the "add meta data to the ACAImage section").
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you don't need to do any filling at all except for imgraws
. For the meta values for each row do:
names = dat.dtype.names
for i, row in ...:
meta = {name: row[name] for name in names if not dat.mask[name][i]}
Technically speaking if dat["IMGSIZE"]
is masked then that entire row is junk and should just be skipped. Using a filled value for that column would lead to undefined behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And probably the whole thing can be nicer as:
# Only keep rows with non-masked IMGSIZE
dat = dat[~dat["IMGSIZE"].mask]
for row, imgraw in zip(dat, imgraws):
imgraw = imgraw.reshape(8, 8)
sz = row["IMGSIZE"]
if ..:
...
meta = {name: row[name] for name in names if not np.any(row.mask[name])}
...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we need to go and find missing data to see which of these filters was needed in general. IIRC IMGSIZE is a calculated quantity from another part of the code, so I didn't see a way that it could have a masked value, for example
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that my 2nd suggestion just works in any case and you don't need to spend time digging. It's all going to be fast enough as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your second suggestion doesn't seem to unmask IMGRAW, and I thought the larger point was not to let the masked array end up as the ACAImage still masked?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was assuming your change imgraws = dat["IMGRAW"].filled(-9999)
. (Note the first sentence in https://github.com/sot/mica/pull/297/files#r1539513631).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. I think I addressed the comments here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
I checked the unit test (including mica/archive/tests/test_aca_l0.py::test_get_l0_images
) and verified that the deprecation warning does not show.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Description
Handle filling aca image array differently
Fixes #293
Interface impacts
Testing
Unit tests
Independent check of unit tests by Javier
Functional tests
No functional testing.