-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add function to filter out ASIC tuning data from mast results #804
Add function to filter out ASIC tuning data from mast results #804
Conversation
Closes #800 |
Hello @bhilbert4, Thank you for updating !
Comment last updated at 2021-10-12 19:30:03 UTC |
According to the JWST keyword dictionary, there is only one TEMPLATE value that is related to ASIC tuning, so no need to look for instrument specific template values. Of course, if the instrument teams had other templates they wanted to ignore, those templates could be added to the new list in constants.py and they would be skipped. |
Testing on the dev server looks good. The dark monitor failed, but due to NIRSpec darks with either bad metadata or in need of better filtering based on array size. This was all after successfully filtering out ASIC tuning data for multiple other apertures. Tests in the bad pixel, bias, and readnoise monitors were also successful. |
@bourque I think this is ready for review. I am a little confused about the RTD failure though. From the raw RTD log, it looks like the test is installing Django version 4.0 (which, when I look around, seems not to exist?) See the Django references here: https://readthedocs.org/api/v2/build/14836853.txt |
Thanks @bhilbert4! I think the RTD failures were fixed in a recent PR; updating the branch appears to have fixed it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bhilbert4 Thanks for this fix. The code changes look good! I just had one suggestion on the location of the exclude_asic_tuning
function -- I will leave it up to you if you want to make a change. Feel free to merge when you're ready.
@@ -99,6 +99,7 @@ | |||
from jwql.database.database_interface import NIRSpecBadPixelQueryHistory, NIRSpecBadPixelStats | |||
from jwql.database.database_interface import FGSBadPixelQueryHistory, FGSBadPixelStats | |||
from jwql.instrument_monitors import pipeline_tools | |||
from jwql.instrument_monitors.common_monitors.dark_monitor import exclude_asic_tuning |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is going to be used in multiple monitors, perhaps it is best to move the function to monitor_utils
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. I remember thinking about this, and then seeing that the bias and readnoise monitor are also importing mast_query_darks from the dark_monitor, and I started thinking I should move both functions to monitor_utils....and then I forgot about it. I'll move both over.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I've moved both functions into monitor_utils.py. There were also two copies of initialize_instrument_monitor(), in monitor_utils.py as well as utils.py, so I removed the latter. This might be worth a quick look just to make sure I didn't screw anything up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Such as a bunch of tests.......Oops.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, tests are all passing again.
Thanks @bhilbert4 for the updates here! I'm going to merge this now. |
This PR adds code to the instrument monitors such that any ASIC tuning data returned by the MAST query will be ignored. ASIC tuning data will have suspect pixel values since readout properties will be non-nominal for these data. We don't want these pixel values contaminating measures of dark current, readnoise, etc.
Files are excluded by checking the 'template' keyword in the dictionaries returned by the MAST query.