pds2isis interpreting zero DN values to NULL by default

[victoronline]

ISIS Version 3.5.2 +

Description
pds2isis appears to be interpreting all 0 DN values as NULL by default for 8 bit images

How to reproduce
Select an 8 bit pds image with zero values. Do not select the "Option to convert pixel values to NULL". Run the application. The output cube will have 0 DN values replaced by NULL values.

Additional context
The image we used was the NAC images for DOY 253 in 2019. The mostly consists of 0 DN values with the exception of a column of values toward the right edge of the image.

The expected behavior is to leave 0 DN values as 0 if the option to convert to NULL is not selected.

[jessemapel]

This is probably due to 0 being the default Null value for 8-bit images in ISIS. If the user does not want to convert any pixels to Null, we'd have to pick an 8-bit number that is not in the image to use as Null.

[gravatite]

In this particular case, there is no value within the 8-bit space that ISIS could use as NULL, we get values from 0-255 from the camera, and there are no special values of any kind. In normal processing, the next step is calibration, and at that point, special values are introduced.

[jessemapel]

@gravatite In that case, the data will have to be converted to something with space for special pixel values, at least 16-bit.

The ISIS Cube format doesn't support images without any special pixel ranges, so I'm not sure if there is a way to have 8-bit cubes without NULL pixels.

As a work around right now, I'd try making the output cube 16-bit and specifying the null pixel range in the negatives.

[victoronline]

Jesse, Is this a recent change in ISIS or was this always the case with 8-bit data? Was this the case for earlier missions too? Some of our users believe that this wasn't an issue before. Would it be possible to arrange a call with you and maybe Stuart to discuss this a bit further?

[jessemapel]

@victoronline I'm not sure if this is new behavior or not. @scsides May have some more insight. We can definitely get on a call with you guys later. Send us an e-mail and we can coordinate it.

[scsides]

@victoronline, the zero value in 8-bit images is hard coded to be the NULL, and has been since ISIS2. As far as we know there have not been any changes that would change the behavior of pds2isis. @jessemapel idea of going to 16 bit is your only way of not treating zeros as NULL, but that said, it will take some tweaking to figure out how to do it without converting the 8 bit NULL to 16 bit NULL.

[jessemapel]

@scsides I am curious if using the +16bit option on the output image along with the nullrange parameters will do the trick here.

[rbeyer]

This is probably due to 0 being the default Null value for 8-bit images in ISIS. If the user does not want to convert any pixels to Null, we'd have to pick an 8-bit number that is not in the image to use as Null.

Yes, but ISIS cubes don't work that way (as you know). In a GeoTIFF, you can specify what value 'nodata' should be. It could be different for every GeoTIFF you make, because it is configurable in the label. With ISIS, the numerical values of the five special pixels (for each of the bit-types) are hard-coded in the software itself. So if you have an 8-bit cube, an 8-bit value of zero will always be interpreted by ISIS as NULL, and for all intents and purposes, ISIS cannot 'see' a value of zero in an 8-bit file.

If you really want to solve this and have user-specifiable per-cube special pixel values, that is something that permeates the ISIS codebase and how it deals with ISIS cube files. You're not going to solve that with a sprint. The solution here is a workflow tweak, and Jesse's suggested 'work around' is really the long term solution. Just convert the image to 16-bit (or whatever the next step in your processing chain needs) one step earlier, when you pull in the data via pds2isis.

[scsides]

@victoronline, @jessemapel, The conversion of special pixels happens at the ISIS::Buffer level, so adding +16bit in pds2isis won't help, I tested it too. So, here are steps to do what I think you want:

pds2isis from=nac8bit.img to=nac8bit.cub
stretch from=nac8bit.cub to=nac16bit.cub+16bit NULL=0

The resulting 16 bit cube should have DNs of zero. Let us know how this goes.

[victoronline]

This issue arose when we tried to use ISIS as a "quick-look" tool to take a peek at the data during a trouble-shooting exercise. We didn't have calibration or kernel information for these images yet but needed to see the raw values coming from the s/c. Unfortunately, since the s/c was still too cold, the expected zero values came across as nulls when looking at the image stats and the image in QView. ISIS is probably not the tool for this purpose since it doesn't totally support 8bit images. Although, I did notice the LIS/HIS behavior for 0 and 255 values for 8bit images is mentioned in the documentation under "SETNULLRANGE", I suggest this behavior for 8bit images be made more explicit in the documentation. Maybe just stating 8bit images are not fully supported or are handled in a special manner. Or maybe mentioning that they should be converted to 16bit if the raw values are intended to be retained. You could also mention the stretch that Stuart suggests in the comment preceding this one. I hope that made sense.

[jlaura]

pds2isis interpreting zero DN values to NULL by default