Missing Camera model (?) for MEX SRC data

[lwellerastro]

ISIS version(s) affected:5.0.2

Description
Mars Express SRC IMG files run through hrsc2isis and spiceinit are successful, however there is no camera/geometry information for the resulting ISIS cubes.

There is no lat/lon information displayed via qview.
camstats reports NULL for everything.
Footprintinit errors with "No lat/lon data found for image in ImagePolygon.cpp at 547".
There's simply no geometric information available.

According to hrsc2isis documentation and git post 3665 we have the capability to work this detector, but maybe I'm reading it wrong?

How to reproduce
Input IMG and output cub files are available under /work/users/lweller/Isis3Tests/MEX_SRC/. One of the files was the topic for post 3665.

hrsc2isis from=H0649_0003_SR2.IMG to=H0649_0003_SR2.cub
spiceinit from=H0649_0003_SR2.cub
camstats from=H0649_0003_SR2.cub linc=10 sinc=10

some output from camstats:
  Group = Latitude
    LatitudeMinimum           = NULL
    LatitudeMaximum           = NULL
    LatitudeAverage           = NULL
    LatitudeStandardDeviation = NULL
  End_Group

  Group = Longitude
    LongitudeMinimum           = NULL
    LongitudeMaximum           = NULL
    LongitudeAverage           = NULL
    LongitudeStandardDeviation = NULL

These data were originally processed via the HPC and IMG's were pulled from various data directories under /work/projects/PhobosMapping/data/MarsExpress/SRC/. We don't have the data available via our local PDS so a copy of what was available from ESA was mirrored locally a couple of years ago. The data had been ingested and spiced at the time and I noticed while looking at those cubes that there was no geometry information on those, so I decided to reprocess some of them from ingestion to ensure I was getting the latest updates to software.

Possible Solution
easy I hope!

Additional context
This is for the internal Phobos project. I will need access to some updated version of isis with fixes as soon possible, otherwise I am unable to work on that project.

[lwellerastro]

Made a comparison of the kernels loaded onto one of the SRC test images and an HRSC image and saw that the HRSC was picking up a more recent IAK than the SRC. Seems odd to me that the SRC wouldn't load the most recent version. Maybe this is the problem?

I copied the test SRC image to a new file and spiced using the more recent IAK and got an error.

spiceinit from=newIAK_H0715_0003_SR2.lev0.cub iak=/usgs/cpkgs/isis3/isis_data/mex/kernels/iak/hrscAddendum004.ti
    Error                     = "Can not find [INS-41220_TRANSX] in text
                                 kernels"

The above image is in the same directory as mentioned in the original post above.
Here's what H0715_0003_SR2.lev0.cub picks up by default:

  Group = Kernels
    NaifIkCode                = -41220
    LeapSecond                = $base/kernels/lsk/naif0012.tls
    TargetAttitudeShape       = $base/kernels/pck/pck00009.tpc
    TargetPosition            = (Table, $base/kernels/spk/de430.bsp,
                                 $base/kernels/spk/mar080.bsp)
    InstrumentPointing        = (Table,
                                 $mex/kernels/ck/ATNM_MEASURED_040101_050101_V-
                                 03.BC, $mex/kernels/fk/MEX_V16.TF)
    Instrument                = $mex/kernels/ik/MEX_HRSC_V03.TI
    SpacecraftClock           = $mex/kernels/sclk/MEX_220209_STEP.TSC
    InstrumentPosition        = (Table,
                                 $mex/kernels/spk/ORMM__040801000000_00088.BSP)
    InstrumentAddendum        = $mex/kernels/iak/hrscsrcAddendum001.ti
    ShapeModel                = Null
    InstrumentPositionQuality = Reconstructed
    InstrumentPointingQuality = Reconstructed
    CameraVersion             = 1
    Source                    = isis
  End_Group

Here's the Kernels group from one of my processed HRSC images that is close in time to the SRC above (/scratch/lweller/Phobos/Lev0/H0756_0000_ND2.lev0.cub):

  Group = Kernels
    NaifIkCode                = -41215
    LeapSecond                = $base/kernels/lsk/naif0012.tls
    TargetAttitudeShape       = $base/kernels/pck/pck00009.tpc
    TargetPosition            = (Table, $base/kernels/spk/de430.bsp,
                                 $base/kernels/spk/mar080.bsp)
    InstrumentPointing        = (Table,
                                 $mex/kernels/ck/ATNM_RECONSTITUTED_00004.BC,
                                 $mex/kernels/fk/MEX_V16.TF)
    Instrument                = $mex/kernels/ik/MEX_HRSC_V03.TI
    SpacecraftClock           = $mex/kernels/sclk/MEX_220209_STEP.TSC
    InstrumentPosition        = (Table,
                                 $mex/kernels/spk/ORMM__040801000000_00088.BSP)
    InstrumentAddendum        = $mex/kernels/iak/hrscAddendum004.ti
    ShapeModel                = Null
    InstrumentPositionQuality = Reconstructed
    InstrumentPointingQuality = Reconstructed
    CameraVersion             = 1
    Source                    = isis
  End_Group

All of the HRSC images pick up Addendum004.ti.

According to /usgs/cpkgs/isis3/isis_data/mex/kernels/ik/MEX_HRSC_V03.TI,
MEX_HRSC_SRC -41220
The instrument ID -41220 is not included in any of the Addendum files.

[scsides]

These aren't two versions of the same file. One is for HRSC (stereo linescaner) the other is for HRSC SRC (Super Res Camera framer).

[lwellerastro]

oh, now I see the difference - just skipped right over that extra src in the SRC addendum file. Darn.

[scsides]

Update: The documentation on HRSC SPICE site is saying that the default SPK for Phobos and Deimos may not be accurate (ISIS' Target SPK). I've run spiceinit using all of the alternatives and none of them allows us to get a target intersection. If I run spiceinit with only NADIR pointing we get body intersections.

I'm currently retesting the alternative SPKs with the whole image to see if we get any intersections. The quick test showed no intersection at the center of the test image. I'm also testing the other available CKs to see if one of them will allow the model to intersect the body.

[lwellerastro]

Huh. Why would the HRSC data of Phobos be ok though - same spacecraft. And picking up the same spacecraft kernel - at least for the two images close in time I compared a few comments back.

[scsides]

Not sure, could be because of the way the HRSC is rotated to acquire the linescan image vs SRC being a snap shot framer. I'm also looking at the rotations for each. Same spacecraft, but different mounting.

[jessemapel]

The HSRC and SRC appear to be a fixed offset from each other so I'm also confused. From the FK at NAIF

                               "J2000" INERTIAL
           +-----------------------------------------------------+
           |                          |                          |
           |<-pck                     |                          |<-pck
           |                          |                          |
           V                          |                          V
       "IAU_MARS"                     |                     "IAU_EARTH"
     MARS BODY-FIXED                  |<-ck               EARTH BODY-FIXED
     ---------------                  |                   ----------------
                                      V
                               "MEX_SPACECRAFT"
           +-----------------------------------
           |
           |<-fixed
           |
           V
    "MEX_HRSC_BASE"
    -------------------------+
           |                 |
           |<-fixed          |<-fixed
           |                 |
           V                 V
    "MEX_HRSC_HEAD"    "MEX_HRSC_SRC"
    ---------------    --------------

[lwellerastro]

Hi, I went ahead and processed all 4826 SRC images of Phobos that I recently downloaded from the archive. I applied general processing as described below expecting lots of failures, but to my surprise, there were some successes - 608. Success means camstats, footprintinit and caminfo did not fail. It does not mean the spice is on the target or easily found in the frame.

Attached is one of the images (left) that made it all of the way through caminfo and the phocube Phase Angle band of the same image (right) showing where the spice ended up. Maybe this is useful? And yes, that is Saturn to the lower left of Phobos in the first image.

I can point you to more successful images if you like. The images in the figure can be found here: /work/users/lweller/Isis3Tests/MEX_SRC/Success/

The following programs were run on it:

hrsc2isis from=/scratch/lweller/Phobos/Raw/HD320_0098_SR2.IMG to=HD320_0098_SR2.lev0.cub 
spiceinit from=HD320_0098_SR2.lev0.cub 
camstats from=HD320_0098_SR2.lev0.cub attach=true format=PVL append=FALSE linc=10 sinc=10 
footprintinit from=HD320_0098_SR2.lev0.cub increaseprecision=FALSE inctype=vertices numvertices=100 maxemission=89 maxincidence=95 
editlab from=HD320_0098_SR2.lev0.cub options=addkey grpname=Instrument keyword=SourceProductId value=HD320_0098_SR2 
caminfo from=HD320_0098_SR2.lev0.cub to=HD320_0098_SR2.lev0.pvl isislabel=true geometry=true originallabel=true statistics=true camstats=true linc=10 sinc=10 uselabel=true

[lwellerastro]

Hi - we heard from Konrad Willner who has worked extensively with these data and he had these comments:

...
However, all the offsets visible in the example are way too large from what I experienced with 
SRC data. Such magnitudes of offsets are realistic for Viking images – where the prediction 
may also be located well outside the field of view of the camera  based on the initial orientation data.

I attach the four screenshots, corresponding to your examples, from our limb fitting software showing 
the bodies predicted limb in red – no shifts or adjustments done to it, only the initial geometric data 
applied. Do achieve this result I used the following kernels.

\begindata


KERNELS_TO_LOAD = ( '/path/spkmerge/mex_all.bsp'

                    '/path/kernels/spk/DE405.BSP'

                    '/path/kernels/spk/MEX_STRUCT_V01.BSP'

                    '/path/special_kernels/MAR097.BSP'

                    '/path/spkmerge/mex_all.bc'

                    '/path/common_kernel//naif0012.tls'

                    '/path/common_kernel//pck00010.tpc'

                    '/path/mars94/kernels/ik/MEX_HRSC_V09.TI'

                    '/path/mars94/kernels/fk/MEX_V16.TF'

                    '/path/mars94/kernels/sclk/MEX_220406_STEP.TSC')

 
Except for the ‘mex_all.*’ kernels, all should be available through PDS or PSA 
(ftp://spiftp.esac.esa.int/data/SPICE/MARSEXPRESS/kernels |
https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS/repos/mars-express/browse  ). 
The Metakernel included in the package from PSA would work too. The ‘mex_all.*’ are just merged 
versions of the SPK and CK single files available for simpler use. If you have troubles getting 
hold of these kernels I could also provide these. Please let me know.

Compared to what my images are picking up via spiceinit, many of the above MEX specific kernels appear to be many versions of ahead. I hope this information proves to be useful.

[scsides]

After working with the HRSC kernels mentioned in comment and comment above there are differences between them and the current kernels in the ISIS data area. On a limited set of images, the new kernels improve the target intersection issue by a few pixels. For some images that had no target intersection even though Phobos was in the image, there was still no target intersection.

The 90 degree rotation observed in the test images that did have a target intersection has been corrected with a change to the ISIS addendum kernel for SRC. The fix involved associating the +X with -sample and +Y with +lines. Since the SRC array is square, this fix is not expected to change the number pixels that intersected the body, only the position of those pixels.

The correction above allows a reasonable target intersection similar to other images supplied by Konrad.

All of the changes until now have been with data files. No software has been changed.

There are DSK shape models available for both Phobos and Deimos

For testing purposes only, the spiceinit command line should look like the following with cwd set above the temp kernel area:

spiceinit from=HDxxxxxxxxx.cub tspk='($base/kernels/spk/de405.bsp, tspk/MAR097.bsp)' ik=ik/MEX_HRSC_V09.ti shape=user model=dsk/PHOBOS_K137_DLR_V01.bds iak= iak/hrscsrcAddendum002.ti

[scsides]

Synopsis of activities to solve the issue of HRSC SCR target intersection not aligning with what the image data (i.e., SRC image -vs- phocube). NOTE: These are a combination of the efforts of Lynn, Brent, Randy, Conrad, Stuart:

FTP site for documentation and SPICE data: https://spiftp.esac.esa.int/data/SPICE/MARS-EXPRESS/kernels/spk/aareadme.txt

Main issues is: The camera model has poor to no target intersection in images where Phobos is visible in many SRC images.

• Pointing is improved on some test images by using ISIS's nadir pointing in spiceinit
• SCR images of Mars have intersections, although they don't line up with HRSC stereo well
• The Target SPKs commented on above improve the pointing of some of the test images
  • Documentation on the different Target SPK's indicates substantial differences with the PDS data for Phobos
  • Tests will need to be run to see which is better for control purposes
• In the camera model as it was there appears to be a 90deg rotation between the visual image data and the phocube data. This is more evident when using the DSK shape model
• There is no distortion model implemented for SRC. Documentation indicates distortion at the corners is less than 1 pixel
• The ISIS data area for SRC has the same Mex CK and SPK data as the FTP site
• The ISIS data area does not contain derived Target SPK from the FTP site
• CK and SPK data for Phobos is contained inside of the CK and SPK files that are also used for Mars
• Looking at the rotation data for Mex the focal plane for SRC was 90 deg off and mirrored right to left from the ISIS TRANS affine xform. The IAK for ISIS was updated to address this.
• Images of Phobos by SRC are usually in a series. Many exposures taken over a short period
  • The SPACECRAFT_CLOCK_START_COUNT for a series (possible not all) are all identical.
  • The START_TIME for a series are different
  • The ISIS camera model was using the SPACECRAFT_CLOCK_START_COUNT to get the center exposure time of the frame
  • The camera model was modified to use the START_TIME. GitHub PR Changed SRC camera and serial number to use start time instead of sta… #4910
  • The ISIS serial numbers for SRC were using the clock time as well and were modified to use the START_TIME
    • This change made existing control networks invalid

At the time of this comment: The PR and the data changes are not in the system. Once testing has been completed this will be done.

[scsides]

Reviews for the PR are in the works. This is a breaking change, so it will be released in the next major version, 8.0, expected release candidate in early Oct, and full release in early Nov.

In order for this work to continue before the 8.0 release, you will need a local build. Someone will need to follow the instructions at, https://github.com/USGS-Astrogeology/ISIS3/wiki/Developing-ISIS3-with-cmake. If help is needed with this process, please start an issue, so the documentation can be improved.

Current process says this is not a breaking change.

[lwellerastro]

@ssides, the work described above included use of kernels that we do not have in the system. Although the camera model needed some changes, some of the spice issues were addressed by using more up to date kernels. Will they be part of the check-in process?

[scsides]

I will move the newer SPICE over to the main data area when the issue is merged into the dev branch. This keeps the continuous integration builds from failing. Basically, both have to happen at the same time.

[rfergason]

What is the status on this PR and related needed efforts (e.g., updating kernels) described above? I see it was merged on April 15, yet there are follow-on comments communicating that reviews for the PR are in the works. I don't understand the connection between these comments. Thanks!

@scsides

[scsides]

The PR was merged, and the data has been put into the main local data area.

[scsides]

There were questions about which of the MEX specific target spk files to use (MAR033_2000-2025.bsp MAR033_HRSC_V03.bsp MAR080.bsp MAR097.bsp MARSAT_ESA_060101_V01.bsp) :

1. ISIS currently uses MAR097.bsp for Mars, but MAR080.bsp for Phobos and Deimos
2. MAR080.bsp in the ISISDATA area and from the ESA ftp site differ.
3. What about the other specialized target SPK files listed above?

Resolution:

1. MAR097.bsp is newer and appears to work well for Phobos and Deimos. There are slight difference in the phocube outputs. spkdiff shows a max position difference of 12.7km. We will make MAR097.bsp the default for the moons.
2. Using spkdiff from NAIF, the position differences between them is zero. It doesn't matter which is used.
3. The documentation for the other SPK files indicates each is better at different times. We will not add these files to the ISISDATA area.

We will change ISIS to use MAR097.bsp for Phobos and Deimos.

These changes have been made in the ISISDATA area