From 1950b10d94ef008d3a530f36e163b9269fcbd0c1 Mon Sep 17 00:00:00 2001 From: Victor Silva Date: Mon, 7 Jun 2021 16:30:09 -0700 Subject: [PATCH 01/10] updated to use time dependent darks including darks for exp zero and also allowed passthru of certain special pixel types instead of nulling them --- isis/src/lro/apps/lronaccal/lronaccal.xml | 180 ++++---- isis/src/lro/apps/lronaccal/main.cpp | 528 +++++++++++++++++++--- 2 files changed, 567 insertions(+), 141 deletions(-) diff --git a/isis/src/lro/apps/lronaccal/lronaccal.xml b/isis/src/lro/apps/lronaccal/lronaccal.xml index ed7d6aebbc..ff66db3f98 100644 --- a/isis/src/lro/apps/lronaccal/lronaccal.xml +++ b/isis/src/lro/apps/lronaccal/lronaccal.xml @@ -8,97 +8,67 @@

- lronaccal performs radiometric corrections to images acquired by the Narrow Angle - Camera aboard the Lunar Reconnaissance Orbiter spacecraft. + lronaccal performs radiometric corrections to images acquired by the Narrow Angle + Camera aboard the Lunar Reconnaissance Orbiter spacecraft. The LRO NAC camera + will make observations simulteously with the HiRise camera.

- The LRO NAC detector has a total of 5064 pixels, divided among an A - channel and a B channel. The pixels alternate between the two channels: - ABABABAB, etc. Images from LROC NAC may or may not include all pixels in the - acquired image. There are special summing modes that are utilized - on-board the spacecraft to average detector pixels to combine them into a - single output pixel value. The value of the ISIS label keyord, - SpatialSumming, indicates the number of samples that were summed and - averaged to result in the pixel values stored in the file. Note that this - will reduce the number of samples in the output image by a factor of at - most the SpatialSumming mode value. + The LRO NAC detector has a total of 5064 pixels, divided among an A + channel and a B channel. The pixels alternate between the two channels: + ABABABAB, etc. Images from LROC NAC may or may not include all pixels in the + acquired image. There are special summing modes that are utilized + on-board the spacecraft to average detector pixels to combine them into a + single output pixel value. The value of the ISIS label keyord, + SpatialSumming, indicates the number of samples that were summed and + averaged to result in the pixel values stored in the file. Note that this + will reduce the number of samples in the output image by a factor of at + most the SpatialSumming mode value.

- The LROC NAC camera has the ability to acquire images of differing sizes in - both line and sample. The starting hardware detector pixel for the - acquired image is specified by the ISIS label keyword, SampleFirstPixel. - The first pixel in the detector is indicated by a value of 0. + The LROC NAC camera has the ability to acquire images of differing sizes in + both line and sample. The starting hardware detector pixel for the + acquired image is specified by the ISIS label keyword, SampleFirstPixel. + The first pixel in the detector is indicated by a value of 0.

- Dark current pixels are taken for each line from the masked pixels - that lie along each edge of the image. -

- -

- If SpatialSumming is 1 the dark current pixels are averaged together then - this average is subtracted from all image pixels. If SpatialSumming is 2, - the dark current pixels for the A and B channel are averaged separately, - then the A channel dark average is subtracted from the A channel image - pixels and the B channel dark average is subtracted from the B channel - image pixels. + Dark current pixels are taken for each line from the masked pixels + that lie along each edge of the image.

- The DN level in an uncalibrated image is the sum of the true signal from the scene, - the bias, the dark current, and random noise in all 3 components. The random noise in - the true signal and dark current is called shot noise and the random noise in the bias - is called read noise. The true signal, bias, and dark current are defined as mean - values so that if the random noise were averaged down to insignificance by taking a - very large number of images and averaging them, the resulting image would be the true - scene, bias, and dark current with no systematic error. That implies the statistical - distribution of the random noise has an average of zero, and therefore the random noise - has both positive and negative values, except for the trivial case of zero random noise. + If SpatialSumming is 1 the dark current pixels are averaged together then + this average is subtracted from all image pixels. If SpatialSumming is 2, + the dark current pixels for the A and B channel are averaged separately, + then the A channel dark average is subtracted from the A channel image + pixels and the B channel dark average is subtracted from the B channel + image pixels.

- The calibration equation is: - 

  reportedDN = ObservedDN - MeanBias - DarkCurrent
+ The dark average produced is dependant on which options are selected. + If the custom dark file option is selected, the provided dark file will be used. - Where: - 
-   ObservedDN = TrueDN + E
-   E is a randomly sampled value from (mu, sigma^2) and mu=0
-   TrueDN is the signal that would be reported in an idealized case of an instrument with zero noise.
-

+ If the image has exposure code of zero, the nearest dark files with exposure + code of zero will be used. -

- Let's look at the case of a calibrated image for which the true signal - is zero, a dark image. In calibration the mean bias and dark current are - subtracted. The random noise term is then randomly sampled from a known - distribution with a mean of zero. Since the distribution has a mean of - zero, values for the random noise can be positive or negative. - Therefore, the addition of random noise to a pixel with true signal near - zero can result in negative DN values. -

+ If UseNearestTimeDarkFile option is selected, the dark file with the nearest + time to the image will be used for dark correction. -

- Negative reported DNs are possible when E < -1 * TrueDN. These are - pixels in a very dark image that happen to have a strongly negative - random noise value. -

+ If UseNearestTimeDarkFilePair option is selected, the pair of dark files that + the image time lies between, will be used, as long as the difference between + dark files doesn't exceed 45 days. If a suitable pair is not found, the + latest dark file taken before the image will be used. + + + pixel_dark_average = + avgDarkLine1_pixel * |darkfile1_time - time| + avgDarkLine2_pixel * |darkfile2_time - time| + / (1.0 * ( |darkFile1_time - time| + |darkFile2_time - time| ) ) -

- Note: ObservedDN and TrueDN both must be greater than or equal to zero. - For ObservedDN, it's because the hardware is not able to report negative - DN values . For TrueDN, it's because radiance and reflectivity cannot be - negative. The dimmest target is one that is completely dark, and for - that target TrueDN = 0. -

-

- If run on a non-spiceinited cube, this program requires access to local mission-specific - SPICE kernels, in order to find the distance between the sun and the target body. - When run on a spiceinited cube, this can be determined using the camera model. - Using a spiceinited cube as input has the advantage of not requiring that local - mission-specific kernels be available. (See spiceinit web=true.)

+ @@ -108,6 +78,13 @@ No longer treat negative DNs differently in the Non-linearity correction. + + Added option for base calibration directory + + + Updated dark calibration to use nearest pair of dark images to calibrate + + @@ -169,30 +146,61 @@ - - + + boolean True - Calibrate using the average dark pixels. + Calibrate using average dark pixels. + + + + DarkFileType + + + + string + + Custom Dark, Nearest Dark, Nearest Dark Pair? - DarkFile + Pair + + + + + - + filename - Default + - Calibrate using the average dark pixels. - + boolean @@ -207,7 +215,7 @@ NonlinearityFile - + filename Default @@ -217,7 +225,7 @@ - + filename Default @@ -228,7 +236,7 @@ - + boolean @@ -240,7 +248,7 @@ FlatfieldFile - + filename Default @@ -251,7 +259,7 @@ - + boolean @@ -266,7 +274,7 @@ RadiometricFile - + string @@ -288,7 +296,7 @@ - + filename Default diff --git a/isis/src/lro/apps/lronaccal/main.cpp b/isis/src/lro/apps/lronaccal/main.cpp index 64dd00e677..a371dcb8b7 100644 --- a/isis/src/lro/apps/lronaccal/main.cpp +++ b/isis/src/lro/apps/lronaccal/main.cpp @@ -9,14 +9,20 @@ find files of those names at the top level of this repository. **/ #include "Isis.h" #include "ProcessByLine.h" #include "SpecialPixel.h" +#include "Message.h" #include "Camera.h" #include "iTime.h" #include "IException.h" #include "TextFile.h" #include "Brick.h" #include "Table.h" +#include "PvlGroup.h" #include "Statistics.h" #include +#include +#include +#include +#include #include using namespace std; @@ -24,7 +30,7 @@ using namespace Isis; // Working functions and parameters void ResetGlobals(); -void CopyCubeIntoArray(QString &fileString, vector &data); +void CopyCubeIntoVector(QString &fileString, vector &data); void ReadTextDataFile(QString &fileString, vector &data); void ReadTextDataFile(QString &fileString, vector > &data); void Calibrate(Buffer &in, Buffer &out); @@ -33,6 +39,11 @@ void CorrectDark(Buffer &in); void CorrectNonlinearity(Buffer &in); void CorrectFlatfield(Buffer &in); void RadiometricCalibration(Buffer &in); +void GetNearestDarkFile(QString fileString, QString &file); +void GetNearestDarkFilePair(QString &fileString, QString &file0, QString &file1); +void GetCalibrationDirectory(QString calibrationType, QString &calibrationDirectory); +void GetWeightedDarkAverages(); +bool AllowedSpecialPixelType(double pixelValue); #define LINE_SIZE 5064 #define MAXNONLIN 600 @@ -40,20 +51,50 @@ void RadiometricCalibration(Buffer &in); #define KM_PER_AU 149597871 #define MASKED_PIXEL_VALUES 8 -vector g_maskedPixelsLeft, g_maskedPixelsRight; +/** + * DarkFileInfo comparison object. + * + * Used for sorting DarkFileInfo objects. Sort first by difference from NAC time + * + */ +struct DarkFileComparison { + int nacTime; + + DarkFileComparison(int nacTime) + { + this->nacTime = nacTime; + } -double g_radianceLeft, g_radianceRight, g_iofLeft, g_iofRight; + // sort dark files by distance from NAC time + bool operator() ( int A, int B) { + if (abs(nacTime - A) < abs(nacTime - B)) + return true; + return false; + } +}; +double g_radianceLeft, g_radianceRight, g_iofLeft, g_iofRight, g_imgTime; double g_exposure; // Exposure duration double g_solarDistance; // average distance in [AU] bool g_summed, g_masked, g_maskedLeftOnly, g_dark, g_nonlinear, g_flatfield, g_radiometric, g_iof, g_isLeftNac; - -vector g_averageDarkLine, g_linearOffsetLine, g_flatfieldLine; - +bool g_nearestDark, g_nearestDarkPair, g_customDark; +vector g_maskedPixelsLeft, g_maskedPixelsRight; +vector g_avgDarkLineCube0, g_avgDarkLineCube1, g_linearOffsetLine, g_flatfieldLine, g_darkTimes, g_weightedDarkTimeAvgs; vector > g_linearityCoefficients; - -// Main moccal routine +Buffer *g_darkCube0, *g_darkCube1; + +/** + * This is the main lronaccal method. Lronaccal is used to calibrate LRO images + * * + * @internal + * @history 2020-01-06 Victor Silva - Added option for base calibration directory + * @history 2020-07-19 Victor Silva - Updated dark calibration to use dark file option + * custom, nearest dark, or nearest dark pair. + * @history 2021-01-09 Victor Silva - Added code to check for exp_code = zero and if so + * then use only exp_code_zero dark files for dark calibration + * + */ void IsisMain() { ResetGlobals(); // We will be processing by line @@ -73,7 +114,8 @@ void IsisMain() { Isis::PvlGroup &inst = lab.findGroup("Instrument", Pvl::Traverse); // Check if it is a NAC image - QString instId = inst["InstrumentId"]; + QString instId = (QString) inst["InstrumentId"]; + instId = instId.toUpper(); if(instId != "NACL" && instId != "NACR") { QString msg = "This is not a NAC image. lrocnaccal requires a NAC image."; throw IException(IException::User, msg, _FILEINFO_); @@ -109,13 +151,15 @@ void IsisMain() { if(iCube->statistics()->Maximum() > 1000) g_maskedLeftOnly = true; - QString darkFile, flatFile, offsetFile, coefficientFile; + QString flatFile, offsetFile, coefficientFile; if(g_masked) { QString maskedFile = ui.GetAsString("MASKEDFILE"); - if(maskedFile.toLower() == "default" || maskedFile.length() == 0) - maskedFile = "$lro/calibration/" + instId + "_MaskedPixels.????.pvl"; + if(maskedFile.toLower() == "default" || maskedFile.length() == 0){ + GetCalibrationDirectory("", maskedFile); + maskedFile = maskedFile + instId + "_MaskedPixels.????.pvl"; + } FileName maskedFileName(maskedFile); if(maskedFileName.isVersioned()) @@ -144,32 +188,78 @@ void IsisMain() { g_maskedPixelsRight.push_back(toInt(maskedPixels[i])); } - if(g_dark) { - darkFile = ui.GetAsString("DARKFILE"); + vector darkFiles; - if(darkFile.toLower() == "default" || darkFile.length() == 0) { - darkFile = "$lro/calibration/" + instId + "_AverageDarks"; + if(g_dark) { + QString darkFileType = ui.GetString("DARKFILETYPE"); + darkFileType = darkFileType.toUpper(); + if (darkFileType == "CUSTOM") { + g_customDark = true; + ui.GetAsString("DARKFILE", darkFiles); + } + else if (darkFileType == "PAIR" || darkFileType == "") + g_nearestDarkPair = true; + else if (darkFileType == "NEAREST"){ + g_nearestDark = true; + } + else { + QString msg = "Error: Dark File Type selection failed."; + throw IException(IException::User, msg, _FILEINFO_); + } + //Options are NEAREST, PAIR, and CUSTOM + if(g_customDark){ + if(darkFiles.size() == 1 && darkFiles[0] != "") { + CopyCubeIntoVector(darkFiles[0], g_avgDarkLineCube0); + } + else { + QString msg = "Custom dark file not provided. Please provide file or choose another option."; + throw IException(IException::User, msg, _FILEINFO_); + } + } + else { + QString darkFile; + g_imgTime = iTime(inst["StartTime"][0]).Et(); + GetCalibrationDirectory("nac_darks", darkFile); + darkFile = darkFile + instId + "_AverageDarks_*T"; + if(g_summed) darkFile += "_Summed"; + // use exp0 dark files if cube's exp_code=0 + Isis::PvlGroup &pvl_archive_group = lab.findGroup("Archive", Pvl::Traverse); + if((int) pvl_archive_group["LineExposureCode"] == 0) + darkFile += "_exp0"; + darkFile += ".????.cub"; + + if(g_nearestDark){ + darkFiles.resize(1); + GetNearestDarkFile(darkFile, darkFiles[0]); + } + else { + darkFiles.resize(2); + GetNearestDarkFilePair(darkFile, darkFiles[0], darkFiles[1]); + //get weigted time avgs + if(g_darkTimes.size() == 2) + GetWeightedDarkAverages(); + } } - CopyCubeIntoArray(darkFile, g_averageDarkLine); } if(g_nonlinear) { offsetFile = ui.GetAsString("OFFSETFILE"); if(offsetFile.toLower() == "default" || offsetFile.length() == 0) { - offsetFile = "$lro/calibration/" + instId + "_LinearizationOffsets"; + GetCalibrationDirectory("", offsetFile); + offsetFile = offsetFile + instId + "_LinearizationOffsets"; if(g_summed) offsetFile += "_Summed"; offsetFile += ".????.cub"; } - CopyCubeIntoArray(offsetFile, g_linearOffsetLine); - + CopyCubeIntoVector(offsetFile, g_linearOffsetLine); coefficientFile = ui.GetAsString("NONLINEARITYFILE"); if(coefficientFile.toLower() == "default" || coefficientFile.length() == 0) { - coefficientFile = "$lro/calibration/" + instId + "_LinearizationCoefficients.????.txt"; + GetCalibrationDirectory("", coefficientFile); + coefficientFile = coefficientFile + instId + "_LinearizationCoefficients.????.txt"; } ReadTextDataFile(coefficientFile, g_linearityCoefficients); } @@ -178,20 +268,22 @@ void IsisMain() { flatFile = ui.GetAsString("FLATFIELDFILE"); if(flatFile.toLower() == "default" || flatFile.length() == 0) { - flatFile = "$lro/calibration/" + instId + "_Flatfield"; - ; + GetCalibrationDirectory("", flatFile); + flatFile = flatFile + instId + "_Flatfield"; if(g_summed) flatFile += "_Summed"; flatFile += ".????.cub"; } - CopyCubeIntoArray(flatFile, g_flatfieldLine); + CopyCubeIntoVector(flatFile, g_flatfieldLine); } if(g_radiometric) { QString radFile = ui.GetAsString("RADIOMETRICFILE"); - if(radFile.toLower() == "default" || radFile.length() == 0) - radFile = "$lro/calibration/NAC_RadiometricResponsivity.????.pvl"; + if(radFile.toLower() == "default" || radFile.length() == 0){ + GetCalibrationDirectory("", radFile); + radFile = radFile + "NAC_RadiometricResponsivity.????.pvl"; + } FileName radFileName(radFile); if(radFileName.isVersioned()) @@ -268,8 +360,20 @@ void IsisMain() { darkColumns += toString(g_maskedPixelsRight[i]); calgrp += darkColumns; } - if(g_dark) - calgrp += PvlKeyword("DarkFile", darkFile); + if(g_dark){ + PvlKeyword darks("DarkFiles"); + darks.addValue(darkFiles[0]); + if(g_nearestDark) + calgrp += PvlKeyword("DarkFileType", "NearestDarkFile"); + else if (g_nearestDarkPair){ + calgrp += PvlKeyword("DarkFileType", "NearestDarkFilePair"); + darks.addValue(darkFiles[1]); + } + else + calgrp += PvlKeyword("DarkFileType", "CustomDarkFile"); + + calgrp += darks; + } if(g_nonlinear) { calgrp += PvlKeyword("NonlinearOffset", offsetFile); calgrp += PvlKeyword("LinearizationCoefficients", coefficientFile); @@ -297,6 +401,10 @@ void IsisMain() { p.EndProcess(); } +/** + * This method resets global variables + * + */ void ResetGlobals() { g_exposure = 1.0; // Exposure duration g_solarDistance = 1.01; // average distance in [AU] @@ -318,13 +426,26 @@ void ResetGlobals() { g_iof = true; g_isLeftNac = true; g_maskedLeftOnly = false; - g_averageDarkLine.clear(); + g_nearestDarkPair = false; + g_nearestDark = false; + g_customDark = false; + g_avgDarkLineCube0.clear(); + g_avgDarkLineCube1.clear(); g_linearOffsetLine.clear(); + g_darkTimes.clear(); + g_weightedDarkTimeAvgs.clear(); g_flatfieldLine.clear(); g_linearityCoefficients.clear(); + g_imgTime = 0.0; } -// Line processing routine +/** + * This method processes buffer by line to calibrate + * + * @param in Buffer to hold 1 line of cube data + * @param out Buffer to hold 1 line of cube data + * + */ void Calibrate(Buffer &in, Buffer &out) { for(int i = 0; i < in.size(); i++) out[i] = in[i]; @@ -345,27 +466,13 @@ void Calibrate(Buffer &in, Buffer &out) { RadiometricCalibration(out); } -void CopyCubeIntoArray(QString &fileString, vector &data) { - Cube cube; - FileName filename(fileString); - if(filename.isVersioned()) - filename = filename.highestVersion(); - if(!filename.fileExists()) { - QString msg = fileString + " does not exist."; - throw IException(IException::User, msg, _FILEINFO_); - } - cube.open(filename.expanded()); - Brick brick(cube.sampleCount(), cube.lineCount(), cube.bandCount(), - cube.pixelType()); - brick.SetBasePosition(1, 1, 1); - cube.read(brick); - data.clear(); - for(int i = 0; i < cube.sampleCount(); i++) - data.push_back(brick[i]); - - fileString = filename.expanded(); -} - +/** + * Read text data file - overloaded method + * + * @param fileString QString + * @param data vector of double + * + */ void ReadTextDataFile(QString &fileString, vector &data) { FileName filename(fileString); if(filename.isVersioned()) @@ -384,6 +491,13 @@ void ReadTextDataFile(QString &fileString, vector &data) { fileString = filename.expanded(); } +/** + * Read the text data file - overloaded method + * + * @param fileString QString + * @param data multi-dimensional vector of double + * + */ void ReadTextDataFile(QString &fileString, vector > &data) { FileName filename(fileString); if(filename.isVersioned()) @@ -409,6 +523,12 @@ void ReadTextDataFile(QString &fileString, vector > &data) { fileString = filename.expanded(); } +/** + * Remove masked offset + * + * @param in Buffer + * + */ void RemoveMaskedOffset(Buffer &in) { int numMasked = MASKED_PIXEL_VALUES; if(g_summed) @@ -451,15 +571,47 @@ void RemoveMaskedOffset(Buffer &in) { } } +/** + * Dark Correction - will find 2 nearest dark files to perform + * dark correction of the pixel being processed + * + * @param in Buffer + * + */ void CorrectDark(Buffer &in) { - for(int i = 0; i < in.size(); i++) { - if(!IsSpecial(in[i])) - in[i] -= g_averageDarkLine[i]; - else + for (int i = 0; i < in.size(); i++) { + if(g_nearestDarkPair && + (!IsSpecial(in[i]) || AllowedSpecialPixelType(in[i])) && + (!IsSpecial(g_avgDarkLineCube0[i]) || AllowedSpecialPixelType(g_avgDarkLineCube0[i])) && + (!IsSpecial(g_avgDarkLineCube1[i]) || AllowedSpecialPixelType(g_avgDarkLineCube1[i])) && + (!IsSpecial(in[i]) || AllowedSpecialPixelType(in[i])) ){ + double w0 = g_weightedDarkTimeAvgs[0]; + double w1 = g_weightedDarkTimeAvgs[1]; + double pixelDarkAvg = (g_avgDarkLineCube0[i]*w0)+(g_avgDarkLineCube1[i]*w1); + + in[i] -= pixelDarkAvg; + + } else if + ((!IsSpecial(g_avgDarkLineCube0[i]) || AllowedSpecialPixelType(g_avgDarkLineCube0[i])) && + (!IsSpecial(in[i]) || AllowedSpecialPixelType(in[i])) ) { + + in[i] -= g_avgDarkLineCube0[i]; + + } + else { + in[i] = Isis::Null; + + } } } +/** + * Correct non-linearity of the pixel being processed + * + * + * @param in Buffer + */ void CorrectNonlinearity(Buffer &in) { for(int i = 0; i < in.size(); i++) { if(!IsSpecial(in[i])) { @@ -489,6 +641,12 @@ void CorrectFlatfield(Buffer &in) { } } +/** + * Radiometric Calibration of the pixel being processed + * + * + * @param in Buffer + */ void RadiometricCalibration(Buffer &in) { for(int i = 0; i < in.size(); i++) { if(!IsSpecial(in[i])) { @@ -510,3 +668,263 @@ void RadiometricCalibration(Buffer &in) { in[i] = Isis::Null; } } + +/** + * This method returns an QString containing the path of an + * LRO calibration directory + * + * @param calibrationType + * @param calibrationDirectory Path of the calibration directory + * + * @internal + * @history 2020-01-06 Victor Silva - Added option for base calibration directory + */ +void GetCalibrationDirectory(QString calibrationType, QString &calibrationDirectory) { + PvlGroup &dataDir = Preference::Preferences().findGroup("DataDirectory"); + QString missionDir = (QString) dataDir["LRO"]; + if(calibrationType != "") + calibrationType += "/"; + + calibrationDirectory = missionDir + "/calibration/" + calibrationType; +} + +/** + * Finds the best dark files for NAC calibration. + * + * GetNearestDarkFile will get the dark file with the + * closest time (before or after) to the image time + * to be used for calibration. + * + * @param fileString String pattern defining dark files to search + * @param file0 Filename of dark file 1 + */ +void GetNearestDarkFile(QString fileString, QString &file) { + FileName filename(fileString); + QString basename = FileName(filename.baseName()).baseName(); // We do it twice to remove the ".????.cub" + // create a regular expression to capture time from filenames + QString regexPattern(basename); + regexPattern.replace("*", "([0-9\\.-]*)"); + QRegExp regex(regexPattern); + // create a filter for the QDir to only load files matching our name + QString filter(basename); + filter.append(".*"); + // get a list of dark files that match our basename + QDir dir( filename.path(), filter ); + vector matchedDarkTimes; + matchedDarkTimes.reserve(dir.count()); + // Loop through all files in the dir that match our basename and extract time + for (unsigned int i=0; i < dir.count(); i++) { + // match against our regular expression + int pos = regex.indexIn(dir[i]); + if (pos == -1) { + continue; // filename did not match basename regex (time contain non-digit) + } + // Get a list of regex matches. Item 0 should be the full QString, item 1 is time. + QStringList texts = regex.capturedTexts(); + if (texts.size() < 1) { + continue; // could not find time + } + // extract time from regex texts + bool timeOK; + int fileTime = texts[1].toInt(&timeOK); + if (!timeOK) { + continue; // time was not a valid numeric value + } + matchedDarkTimes.push_back(fileTime); + } + // sort the files by distance from nac time + DarkFileComparison darkComp((int)g_imgTime); + sort(matchedDarkTimes.begin(), matchedDarkTimes.end(), darkComp); + int darkTime = matchedDarkTimes[0]; + int fileTimeIndex = fileString.indexOf("*T"); + file = fileString; + file.replace(fileTimeIndex, 1, toString(darkTime)); + CopyCubeIntoVector(file, g_avgDarkLineCube0); +} + +/** + * Finds the best dark files for NAC calibration. + * + * GetNearestDarkFilePair will get the average between the two darks files + * that the image lies between (time-wise). + + * If this pair is not found, the nearest dark file will be used + * for calibration. + * + * @param fileString String pattern defining dark files to search (ie. lro/calibration/nac_darks/NAC*_AverageDarks_*T_.????.cub) + * @param file0 Filename of dark file 1 + * @param file1 Filename of dark file 2 + */ +void GetNearestDarkFilePair(QString &fileString, QString &file0, QString &file1) { + FileName filename(fileString); + QString basename = FileName(filename.baseName()).baseName(); // We do it twice to remove the ".????.cub" + // create a regular expression to capture time from filenames + QString regexPattern(basename); + regexPattern.replace("*", "([0-9\\.-]*)"); + QRegExp regex(regexPattern); + // create a filter for the QDir to only load files matching our name + QString filter(basename); + filter.append(".*"); + // get a list of dark files that match our basename + QDir dir( filename.path(), filter ); + vector matchedDarkTimes; + matchedDarkTimes.reserve(dir.count()); + if (dir.count() < 1){ + QString msg = "Could not find any dark file of type " + filter + ".\n"; + throw IException(IException::User, msg, _FILEINFO_); + } + // Loop through all files in the dir that match our basename and extract time + for (unsigned int i=0; i < dir.count(); i++) { + // match against our regular expression + int pos = regex.indexIn(dir[i]); + if (pos == -1) { + continue; // filename did not match basename regex (time contain non-digit) + } + // Get a list of regex matches. Item 0 should be the full QString, item 1 + // is time. + QStringList texts = regex.capturedTexts(); + if (texts.size() < 1) { + continue; // could not find time + } + // extract time from regex texts + bool timeOK; + int fileTime = texts[1].toInt(&timeOK); + if (!timeOK) { + continue; // time was not a valid numeric value + } + matchedDarkTimes.push_back(fileTime); + } + // sort the files by distance from nac time + DarkFileComparison darkComp((int)g_imgTime); + sort(matchedDarkTimes.begin(), matchedDarkTimes.end(), darkComp); + + int fileTimeIndex = fileString.indexOf("*T"); + int t0 = 0; + int t1 = 0; + //Let's find the first time before the image + for(size_t i = 0; i < matchedDarkTimes.size(); i++){ + if(matchedDarkTimes[i] <= (int)g_imgTime){ + t0 = matchedDarkTimes[i]; + break; + } + } + //Let's find the second time + for (size_t i = 0; i < matchedDarkTimes.size(); i++) { + if (matchedDarkTimes[i] >= (int)g_imgTime) { + t1 = matchedDarkTimes[i]; + break; + } + } + if((t0 && t1) && (t0!=t1)){ + int timeDayDiff = abs(t1 -t0)/86400.0; + + //check time range between darks is within 45 day window + if (timeDayDiff < 0 || timeDayDiff > 45) { + QString msg = "Could not find a pair of dark files within 45 day range that includes the image [" + basename + "]. Check to make sure your set of dark files is complete.\n"; + throw IException(IException::User, msg, _FILEINFO_); + } + else { + file0 = fileString; + file0.replace(fileTimeIndex, 1, toString(t0)); + CopyCubeIntoVector(file0, g_avgDarkLineCube0); + g_darkTimes.push_back(t0); + file1 = fileString; + file1.replace(fileTimeIndex, 1, toString(t1)); + CopyCubeIntoVector(file1, g_avgDarkLineCube1); + g_darkTimes.push_back(t1); + } + } + else { + g_nearestDark = true; + g_nearestDarkPair = false; + int darkTime = matchedDarkTimes[0]; + file0 = fileString; + file0.replace(fileTimeIndex, 1, toString(darkTime)); + CopyCubeIntoVector(file0, g_avgDarkLineCube0); + g_darkTimes.push_back(darkTime); + } +} + +/** + * This method copies cube into vector + * LRO calibration directory + * + * @param fileString QString pointer + * @param data vector of double + * + */ +void CopyCubeIntoVector(QString &fileString, vector &data) { + Cube cube; + FileName filename(fileString); + if(filename.isVersioned()) + filename = filename.highestVersion(); + if(!filename.fileExists()) { + QString msg = fileString + " does not exist."; + throw IException(IException::User, msg, _FILEINFO_); + } + cube.open(filename.expanded()); + Brick brick(cube.sampleCount(), cube.lineCount(), cube.bandCount(), cube.pixelType()); + brick.SetBasePosition(1, 1, 1); + cube.read(brick); + data.clear(); + for(int i = 0; i < cube.sampleCount(); i++) + data.push_back(brick[i]); + + fileString = filename.expanded(); + + if(data.empty()){ + QString msg = "Copy from + " + fileString + " into vector failed."; + throw IException(IException::User, msg, _FILEINFO_); + } + +} +/** + * Allow special pixel types + * + * @param pixelValue double + * + * @return bool + * + */ +bool AllowedSpecialPixelType(double pixelValue) { + bool result = false; + result = result || IsHisPixel(pixelValue); + result = result || IsLisPixel(pixelValue); + result = result || IsHrsPixel(pixelValue); + result = result || IsLrsPixel(pixelValue); + return result; +} + +/** + * Get weighted time average for calculating pixel dark + * average + * + * @param w0 double Weighted Time Average for dark file + * @param w1 double Weighted time Average for dark file + * + */ +void GetWeightedDarkAverages() { + + int iTime = (int)g_imgTime; + int t0 = 0; + int t1 = 0; + + if (!g_darkTimes.empty()){ + if (g_darkTimes.size() == 2){ + t0 = g_darkTimes[0]; + t1 = g_darkTimes[1]; + double weight0 = + (( t1!=iTime ) * ( (t1 > iTime ) * ( t1 - iTime) )) + / (((( t1!=iTime ) * ( (t1 > iTime ) * ( t1 - iTime) )) + + (( t0!=iTime ) * ( (t0 < iTime ) * ( iTime - t0) )) ) * 1.0); + + double weight1 = (( t0!=iTime ) * ( (t0 < iTime ) * ( iTime - t0) )) + / (((( t1!=iTime ) * ( (t1 > iTime ) * ( t1 - iTime) )) + + (( t0!=iTime ) * ( (t0 < iTime ) * ( iTime - t0) )) ) * 1.0); + + g_weightedDarkTimeAvgs.clear(); + g_weightedDarkTimeAvgs.push_back(weight0); + g_weightedDarkTimeAvgs.push_back(weight1); + } + } +} From c4b17c4e7165936d29586471f2016b45cf15a515 Mon Sep 17 00:00:00 2001 From: Victor Silva Date: Mon, 4 Oct 2021 11:41:24 -0700 Subject: [PATCH 02/10] Updated lronaccal.xml documentation Updated documentation to provide more clarity regarding dark file correction/calibration options available to user. --- isis/src/lro/apps/lronaccal/lronaccal.xml | 28 ++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/isis/src/lro/apps/lronaccal/lronaccal.xml b/isis/src/lro/apps/lronaccal/lronaccal.xml index ff66db3f98..c2bc5000d0 100644 --- a/isis/src/lro/apps/lronaccal/lronaccal.xml +++ b/isis/src/lro/apps/lronaccal/lronaccal.xml @@ -84,6 +84,9 @@ Updated dark calibration to use nearest pair of dark images to calibrate + + Updated dark calibration documentation to provide more clarity to dark file options + @@ -155,6 +158,9 @@ Calibrate using average dark pixels. + Calibrate using average dark pixels. This is also referred to + as dark correction. These dark calibration/correction files + are included in the data directory (lro/calibration/nac_darks). DarkFileType @@ -162,27 +168,47 @@ string - Custom Dark, Nearest Dark, Nearest Dark Pair? + Calibrate using average dark pixels from + Nearest Single, Pair, or Custom dark file(s). + There are three options for dark calibration/correction. + + PAIR: + The pair of dark files that the image time lies between + will be used as long as the difference between dark files + doesn't exceed 45 days. If a suitable pair is not found, the + latest dark file taken before the image will be used. + + NEAREST: + The dark file with the nearest time to the image will be used + for dark correction. + + CUSTOM: + User can choose a custom dark file by + selecting "CUSTOM" and selecting custom file on user's local machine. + Pair