Bad pixel removal in master()

[oabsil]

Hi @henry-ngo

Using your updated master() function, I've run into a problem when calling:
mflat = masterFlat(filelist_flat, save=save, path_output=path_output, verbose=True, filtering=5)

With this call, NIRC2_Preprocessing crashes at line 502
--> 502 mimage=cube_fix_badpix_isolated(mimage,bpm_mask=bad_pix_map)
and complains about mimage not being a cube:

---

/Users/oabsil/Python/vip/vip/calib/badpixremoval.pyc in cube_fix_badpix_isolated(array, bpm_mask, sigma_clip, num_neig, size, protect_mask, radius, verbose, debug)
149 """
150 if not array.ndim == 3:
--> 151 raise TypeError('Array is not a 3d array or cube')
152 if size%2==0:
153 raise TypeError('Size of the median blur kernel must be an odd integer')

TypeError: Array is not a 3d array or cube

---

My images are indeed no cubes. The function frame_fix_badpix_isolated() should thus rather be used at line 502.

I'm surprised that this has not been reported by anyone yet. Probably I'm doing something wrong. Are the flats supposed to be cubes?

Cheers,
Olivier

[oabsil]

Well, seems to me that there is another problem later on, when using the function applyFlat():

---

/Users/oabsil/Python/NIRC2_Preprocessing/NIRC2_Preprocessing.py in applyFlat(fileList, path_mflat, header, display, save, verbose, full_output, path_output)
662 #processed = raw/mflat
663 #processed_all[filepath] = raw/mflat
--> 664 processed_all_cube[i,:,:] = raw/mflat
665
666

ValueError: could not broadcast input array from shape (2,1024,1024) into shape (1024,1024)

---

The problem is that mflat is a tuple of arrays now, if I'm not mistaken. Probably I'm doing something wrong, but in any case, it would be good to update the "NIRC2_Preprocessing_ streamlined" ipython notebook to have an example of how to use the update version of NIRC2_Preprocessing.

Cheers,
Olivier

[henry-ngo]

Thanks @oabsil ! I just want to let you know I am looking into this issue today.

[henry-ngo]

Hi @oabsil

For the issue at line 502 (regarding input not being a cube), I am not sure why you have this error. Is it possible that there is a difference in the versions of our VIP packages? My copy shows the last update on Dec 15, 2016 (matches the github version I think) and the check on line 150 doesn't exist in my version! Instead, the dimensionality check is on line 69 and reads:

    if array.ndim>3 or array.ndim<2:
        raise TypeError('Array is not a 2d or 3d array')

which means this function should accept both "frames" and "cubes". To answer the other question, no, flats should not be cubes.

For the issue on line 664, you are right that the output of the function masterFlat() is a tuple of arrays now, not just one single array. It returns one, two, or three arrays depending on whether the header and bpm keyword parameters are set to True. That is, if you want the full output, I would call this function in this way:

    mflat,headers,bad_pix_map = masterFlat(filelist_flat, header=True, bpm=True, verbose=False, ...)

I will update the tutorial to reflect this. The tricky part was that bpm is set to True by default, so the old function call in the tutorial should create two outputs.

[oabsil]

Hi @henry-ngo

Thanks a lot for your answers, and for updating the notebook. Regarding the first issue, this is very strange as the dimensionality check you're referring to has been removed from the badpixelremoval.py file in VIP on 2016 Sept 23 (from what I can see on the commit history). I'll try to clarify that with @CARLGOGO , and will let you know if there is any action to take on the NIRC2_Preprocessing side.

Cheers,
Olivier.