Add starcat_date to observation parameters + fix caching and minor issues #221

taldcroft · 2022-04-07T21:52:19Z

Description

Add starcat_date to the parameters stored in observations.

Other things along the way:

Noticed that the clear_caches was missing the OBSERVATIONS cache, so that was fixed.
Noticed that the result from get_observations was a reference to the original params dicts in cache, so this allowed for shenanigans if the user played with those outputs. This, of course, came up in testing and took quite a while to figure out.
Added some clear_cache() commands in certain tests to ensure independence of results from test order.
Updated to run most of the get_observations or get_starcats test using the flight scenario so they will run on GRETA.

Also includes #225 and #226.

Fixes #220

Interface impacts

Adds a new key starcat_date to the dict returned by get_observations
Changes the star catalog date from being the obs_start (start of NPNT) to starcat_date (commanded star catalog time).
Requires new versions of cmds2.pkl and cmds2.h5. I've generated these and will place them on HEAD and GRETA. The new data files are expected to be compatible with the release 5.9.2 version in 2022.3 but I'll check that.

Running the 5.9.2 unit tests with the new data files gives 3 failures, all expected:

=================================================== short test summary info ====================================================
FAILED kadi/commands/tests/test_commands.py::test_commands_create_archive_regress[2] - AssertionError: assert False
FAILED kadi/commands/tests/test_commands.py::test_get_observations_by_obsid_single - AssertionError: assert [{'manvr_star...2...
FAILED kadi/commands/tests/test_commands.py::test_get_observations_by_obsid_multi - AssertionError: assert [{'manvr_star...00...
======================================== 3 failed, 50 passed, 103 deselected in 31.06s =========================================

Testing

Unit tests

Mac

Independent check of unit tests by Jean

Linux

Functional tests

Jean functionally confirmed the unique option to get_starcats_as_table with this query that gets the two catalogs (same catalog) of an ACIS undercover series.

In [5]: get_starcats_as_table(start='2020:366:17:04:20.000', stop='2020:366:23:00:00.000')
Out[5]: 
<Table length=22>
obsid      starcat_date      slot  idx      id    type  sz         mag          maxmag          yang                zang         dim   res  halfw
int64         str21         int64 int64   int64   str3 str3      float64       float64        float64             float64       int64 int64 int64
----- --------------------- ----- ----- --------- ---- ---- ----------------- --------- ------------------- ------------------- ----- ----- -----
24882 2020:366:17:04:20.000     0     1         1  FID  8x8               7.0       8.0 -1174.9607093061636  -468.1481719926585     1     1    25
24882 2020:366:17:04:20.000     1     2         3  FID  8x8               7.0       8.0 -1177.6232024219992   561.4018012757298     1     1    25
24882 2020:366:17:04:20.000     2     3         4  FID  8x8               7.0       8.0  1223.4393109616512   563.8018390366224     1     1    25
24882 2020:366:17:04:20.000     3     4 552075520  BOT  6x6 8.885130882263184 10.390625  -912.6106246537174  -703.8856678520224     8     1    60
24882 2020:366:17:04:20.000     4     5 552086952  BOT  6x6 8.897590637207031  10.40625 -1510.7731283644346  -2392.854408253782     8     1    60
24882 2020:366:17:04:20.000     5     6 552206400  BOT  6x6 8.648664474487305  10.15625  -84.90442138289295  -2367.635642290802    12     1    80
24882 2020:366:17:04:20.000     6     7 552206936  BOT  6x6 6.875301837921143     8.375  2104.7955129097486  1092.1642810692815    16     1   100
24882 2020:366:17:04:20.000     7     8 552213664  BOT  6x6  8.73717975616455 10.234375    1073.65803292352  -944.4668937469213    12     1    80
24882 2020:366:17:04:20.000     0     9 552217088  ACQ  6x6 8.083139419555664   9.59375  168.79558020035054 -2223.1544736547244    12     1    80
24882 2020:366:17:04:20.000     1    10 552081976  ACQ  6x6  9.25229549407959     10.75 -1094.5981762913334  1597.2393211928293     8     1    60
24882 2020:366:17:04:20.000     2    11 552084112  ACQ  6x6  9.63385009765625 11.140625  -782.1356873895911  -292.7544209467494     8     1    60
62647 2020:366:17:04:20.000     0     1         1  FID  8x8               7.0       8.0 -1174.9607093061636  -468.1481719926585     1     1    25
62647 2020:366:17:04:20.000     1     2         3  FID  8x8               7.0       8.0 -1177.6232024219992   561.4018012757298     1     1    25
62647 2020:366:17:04:20.000     2     3         4  FID  8x8               7.0       8.0  1223.4393109616512   563.8018390366224     1     1    25
62647 2020:366:17:04:20.000     3     4 552075520  BOT  6x6 8.885130882263184 10.390625  -912.6106246537174  -703.8856678520224     8     1    60
62647 2020:366:17:04:20.000     4     5 552086952  BOT  6x6 8.897590637207031  10.40625 -1510.7731283644346  -2392.854408253782     8     1    60
62647 2020:366:17:04:20.000     5     6 552206400  BOT  6x6 8.648664474487305  10.15625  -84.90442138289295  -2367.635642290802    12     1    80
62647 2020:366:17:04:20.000     6     7 552206936  BOT  6x6 6.875301837921143     8.375  2104.7955129097486  1092.1642810692815    16     1   100
62647 2020:366:17:04:20.000     7     8 552213664  BOT  6x6  8.73717975616455 10.234375    1073.65803292352  -944.4668937469213    12     1    80
62647 2020:366:17:04:20.000     0     9 552217088  ACQ  6x6 8.083139419555664   9.59375  168.79558020035054 -2223.1544736547244    12     1    80
62647 2020:366:17:04:20.000     1    10 552081976  ACQ  6x6  9.25229549407959     10.75 -1094.5981762913334  1597.2393211928293     8     1    60
62647 2020:366:17:04:20.000     2    11 552084112  ACQ  6x6  9.63385009765625 11.140625  -782.1356873895911  -292.7544209467494     8     1    60

In [6]: get_starcats_as_table(start='2020:366:17:04:20.000', stop='2020:366:23:00:00.000', unique=True)
Out[6]: 
<Table length=11>
obsid      starcat_date      slot  idx      id    type  sz         mag          maxmag          yang                zang         dim   res  halfw
int64         str21         int64 int64   int64   str3 str3      float64       float64        float64             float64       int64 int64 int64
----- --------------------- ----- ----- --------- ---- ---- ----------------- --------- ------------------- ------------------- ----- ----- -----
24882 2020:366:17:04:20.000     0     1         1  FID  8x8               7.0       8.0 -1174.9607093061636  -468.1481719926585     1     1    25
24882 2020:366:17:04:20.000     1     2         3  FID  8x8               7.0       8.0 -1177.6232024219992   561.4018012757298     1     1    25
24882 2020:366:17:04:20.000     2     3         4  FID  8x8               7.0       8.0  1223.4393109616512   563.8018390366224     1     1    25
24882 2020:366:17:04:20.000     3     4 552075520  BOT  6x6 8.885130882263184 10.390625  -912.6106246537174  -703.8856678520224     8     1    60
24882 2020:366:17:04:20.000     4     5 552086952  BOT  6x6 8.897590637207031  10.40625 -1510.7731283644346  -2392.854408253782     8     1    60
24882 2020:366:17:04:20.000     5     6 552206400  BOT  6x6 8.648664474487305  10.15625  -84.90442138289295  -2367.635642290802    12     1    80
24882 2020:366:17:04:20.000     6     7 552206936  BOT  6x6 6.875301837921143     8.375  2104.7955129097486  1092.1642810692815    16     1   100
24882 2020:366:17:04:20.000     7     8 552213664  BOT  6x6  8.73717975616455 10.234375    1073.65803292352  -944.4668937469213    12     1    80
24882 2020:366:17:04:20.000     0     9 552217088  ACQ  6x6 8.083139419555664   9.59375  168.79558020035054 -2223.1544736547244    12     1    80
24882 2020:366:17:04:20.000     1    10 552081976  ACQ  6x6  9.25229549407959     10.75 -1094.5981762913334  1597.2393211928293     8     1    60
24882 2020:366:17:04:20.000     2    11 552084112  ACQ  6x6  9.63385009765625 11.140625  -782.1356873895911  -292.7544209467494     8     1    60

Tom generated new commands archive data files.

taldcroft · 2022-04-08T10:43:31Z

Although the updated data files are compatible with the code, some unit tests in the release will fail with the new files. For testing I put updated files in ~aldcroft/tmp/kadi/ on HEAD.

taldcroft · 2022-04-13T10:25:47Z

@javierggt - for testing you need to copy the new cmds2.* files I referenced above into a local directory and then point kadi to them via export KADI=<local-kadi-dir>. If you are using events then you need to link/copy events3.db3 as well.

You might have noticed that it seems to work for recent obsids, which is because those are being created in memory on-the-fly and thus had the starcat_date key included.

javierggt · 2022-04-13T16:51:20Z

I started thinking how to make functional tests. For that purpose I created a branch of agasc that includes a function get_star_observations in agasc.supplement.magnitudes.star_obs_catalogs. See here.

Using the commands-v2 branch of agasc and the branch in this PR, I did the following:

from agasc.supplement.magnitudes import star_obs_catalogs
star_obs_catalogs.load()
stars_obs_v1 = star_obs_catalogs.STARS_OBS
stars_obs_v2 = star_obs_catalogs.get_star_observations()

These star-observations should be equivalent or close enough... A simple comparison of OBSIDs and agasc IDs in both:

len(stars_obs_v1): 207219
len(stars_obs_v2): 376708

len(obsid_v1): 32754
len(obsid_v2): 34617
+---------------------|-----------------|---------------------+
|                     | in obsid_v1     | not in obsid_v1     |
+---------------------|-----------------|---------------------+
|     in obsid_v2     |           32584 |                2033 |
| not in obsid_v2     |             170 |                     |
+---------------------|-----------------|---------------------+

len(agasc_id_v1): 87587
len(agasc_id_v2): 90263
+---------------------|-----------------|---------------------+
|                     | in agasc_id_v1  | not in agasc_id_v1  |
+---------------------|-----------------|---------------------+
|     in agasc_id_v2  |           87561 |                2702 |
| not in agasc_id_v2  |              26 |                     |
+---------------------|-----------------|---------------------+

I conclude that the code runs and the starcat_date seem to agree. The commands-v2 version seems to produce more star-observations, which is not specific to this PR.

NOTE: these numbers seem not right to me yet. I am taking a closer look and will update once I know better.

javierggt · 2022-04-13T21:22:34Z

Independently of the numbers above, when I run the following:

from astropy.table import Table
from kadi import commands
commands.conf.commands_version = '2'
observations = Table(commands.get_observations())
np.count_nonzero(observations['starcat_date'].mask)

I see 836 observations out of 40232 that have no starcat_date. Would that be expected?

taldcroft · 2022-04-14T00:29:50Z

I see 836 observations out of 40232 that have no starcat_date. Would that be expected?

This seems reasonable to me. There are normal reasons for this to happen:

Segmented maneuvers, which I'm guessing is represented here: https://icxc.harvard.edu/mp/mplogs/2019/DEC0219/oflsa/starcheck.html#obsid47655
Undercover observations
Old-style dark cals where NMM transitions were in the loads
Any gyro hold activity
Safing action like NSM generates an observation with no star catalog

It's good to look into these corner cases to find problems. I suspect that with deep digging that there are some oddball cases that are just wrong, but investigating many of them is very time consuming.

One thing is this:

(ska3) ➜  ~ find ~/ska/data/mpcrit1/mplogs/*/*/ofls/ -name 'starcheck.txt' | xargs grep 'No star catalog for obsid' | wc
     993   13527  142509

So the 836 you found seems in the right ballpark. I suspect my grep includes obsids that were never run, hence the overcount.

javierggt

I'm ok with these changes, which now include PR #222 and will include PRs #225 and PR #226. I assume this PR will wait until those are merged. Perhaps the title and description should be changed to include them, or are we certain that those PR merges will have the commit message that reads "Merge pull request..."? There is no such commit after 222 was merged.

The only thing I thought when looking at the code was that the cache is going to be cleared whenever one runs tests, but I guess there is no trivial way to fix that.

I ran the unit tests and some functional tests by reproducing the process that will be followed in the agasc supplement update.

taldcroft · 2022-04-14T15:05:53Z

I just updated the description to note that #225 and #226 are included.

taldcroft · 2022-04-19T12:38:11Z

@javierggt - I did the comparison of the AGASC star observations to what comes from kadi commands v2 and found not too much actual difference. This comparison uses the MP_STARCAT time as the key. There is a lot of difference in obsids because AGASC uses the originally commanded obsid while kadi uses the as-run obsid, which is different after SCS-107 runs. The takeaway is reinforcing that obsid is not useful in this context since there are different possible definitions.

Also, kadi starts around 2002:001 while AGASC starts in 2003:001. That 2003:001 was arbitrary and conservative, but I think we can push it back to the 2002 date in kadi.

See https://gist.github.com/taldcroft/d870443430466a96a8d5b2cbe5c1faf3. I find just 8 star catalogs that are in kadi v2 and not in AGASC, and zero the other way round.

jeanconn · 2022-04-20T16:08:33Z

Regarding the 836 without star catalogs, they look to all have npnt_enab == False, which at least seems consistent. Regarding

Segmented maneuvers, which I'm guessing is represented here: https://icxc.harvard.edu/mp/mplogs/2019/DEC0219/oflsa/starcheck.html#obsid47655

Those were actually planned gyro holds (I'm forgetting why but from the load review notes)

· 3 Gyro Holds comprised of 5 attitudes each of which has an earth boresight violation. They start at approximately:
o 2019:335:22:00
o 2019:338:13:00
o 2019:341:04:30

I more strict "segmented maneuver" can be seen at something like:

https://icxc.harvard.edu/mp/mplogs/2002/APR0302/oflsc/starcheck.html#obsid2118

And 2118 has what I would expect in the kadi observations (one no-starcat observation, one starcat observation, each with right attitude and timing that seems reasonable).

jeanconn · 2022-04-20T19:49:16Z

kadi/commands/observations.py

+    The ``unique`` parameter can be set to ``True`` to only return unique star
+    catalog entries. There are numerous instances of a single commanded star
+    catalogs that is associated with two ObsIDs, for instance ACIS undercover
+    observations. To get only the first one, set ``unique=True``.


For the unique flag, the first undercover I tried did not line up with expectations:

In [29]: get_starcats_as_table(start='2020:366:17:04:20.000', stop='2020:366:23:00:00.000') Out[29]: <Table length=22> obsid starcat_date slot idx id type sz mag maxmag yang zang dim res halfw int64 str21 int64 int64 int64 str3 str3 float64 float64 float64 float64 int64 int64 int64 ----- --------------------- ----- ----- --------- ---- ---- ----------------- --------- ------------------- ------------------- ----- ----- ----- 24882 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 24882 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 24882 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 24882 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 24882 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 24882 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 24882 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 24882 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 24882 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 24882 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 24882 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60 24882 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 24882 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 24882 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 24882 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 24882 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 24882 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 24882 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 24882 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 24882 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 24882 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 24882 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60

In [30]: get_starcats_as_table(start='2020:366:17:04:20.000', stop='2020:366:23:00:00.000', unique=True) Out[30]: <Table length=11> obsid starcat_date slot idx id type sz mag maxmag yang zang dim res halfw int64 str21 int64 int64 int64 str3 str3 float64 float64 float64 float64 int64 int64 int64 ----- --------------------- ----- ----- --------- ---- ---- ----------------- --------- ------------------- ------------------- ----- ----- ----- 24882 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 24882 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 24882 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 24882 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 24882 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 24882 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 24882 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 24882 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 24882 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 24882 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 24882 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60

Yes, unique reduced this to one set of entries. I just thought the second set of entries would have obsid 62647 in the table.

This looks like something else that is behaving a little bit oddly with caching (and now my cache has a different state / obsid.

In [10]: with conf.set_temp('cache_starcats', True): ...: print(get_starcats_as_table(start='2020:366:17:04:20.000', stop='2020:366:23:00:00.000')) ...: obsid starcat_date slot idx id type sz mag maxmag yang zang dim res halfw ----- --------------------- ---- --- --------- ---- --- ----------------- --------- ------------------- ------------------- --- --- ----- 62647 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 62647 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 62647 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 62647 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 62647 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 62647 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 62647 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 62647 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 62647 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 62647 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 62647 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60 62647 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 62647 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 62647 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 62647 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 62647 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 62647 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 62647 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 62647 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 62647 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 62647 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 62647 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60 In [11]: with conf.set_temp('cache_starcats', False): ...: print(get_starcats_as_table(start='2020:366:17:04:20.000', stop='2020:366:23:00:00.000')) ...: obsid starcat_date slot idx id type sz mag maxmag yang zang dim res halfw ----- --------------------- ---- --- --------- ---- --- ----------------- --------- ------------------- ------------------- --- --- ----- 24882 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 24882 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 24882 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 24882 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 24882 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 24882 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 24882 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 24882 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 24882 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 24882 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 24882 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60 24882 2020:366:17:04:20.000 0 1 1 FID 8x8 7.0 8.0 -1174.9607093061636 -468.1481719926585 1 1 25 24882 2020:366:17:04:20.000 1 2 3 FID 8x8 7.0 8.0 -1177.6232024219992 561.4018012757298 1 1 25 24882 2020:366:17:04:20.000 2 3 4 FID 8x8 7.0 8.0 1223.4393109616512 563.8018390366224 1 1 25 24882 2020:366:17:04:20.000 3 4 552075520 BOT 6x6 8.885130882263184 10.390625 -912.6106246537174 -703.8856678520224 8 1 60 24882 2020:366:17:04:20.000 4 5 552086952 BOT 6x6 8.897590637207031 10.40625 -1510.7731283644346 -2392.854408253782 8 1 60 24882 2020:366:17:04:20.000 5 6 552206400 BOT 6x6 8.648664474487305 10.15625 -84.90442138289295 -2367.635642290802 12 1 80 24882 2020:366:17:04:20.000 6 7 552206936 BOT 6x6 6.875301837921143 8.375 2104.7955129097486 1092.1642810692815 16 1 100 24882 2020:366:17:04:20.000 7 8 552213664 BOT 6x6 8.73717975616455 10.234375 1073.65803292352 -944.4668937469213 12 1 80 24882 2020:366:17:04:20.000 0 9 552217088 ACQ 6x6 8.083139419555664 9.59375 168.79558020035054 -2223.1544736547244 12 1 80 24882 2020:366:17:04:20.000 1 10 552081976 ACQ 6x6 9.25229549407959 10.75 -1094.5981762913334 1597.2393211928293 8 1 60 24882 2020:366:17:04:20.000 2 11 552084112 ACQ 6x6 9.63385009765625 11.140625 -782.1356873895911 -292.7544209467494 8 1 60

So if the starcats are just in the cache with one key, I would assume that get_starcats or get_starcats_as_table would need to hold on to the obsid to assign it in a way that makes sense for the undercover in this special case. This is not new for this PR but I think is something I missed in the PR #226 review.

Awesome testing @jeanconn thanks!!

I think I just missed putting the obsid in the cache key. I'm actually not sure the source (load name) really matters but it doesn't hurt. See c4775e8. Does this key seem unique to you now?

Thanks! I wasn't sure that we actually needed to put the catalog in the cache twice for these observations that share catalog? But using a unique keys does seem simpler than pulling out the one star catalog twice and labeling it again with obs['obsid'] as part of the de-caching in get_starcats.

taldcroft · 2022-04-20T21:22:49Z

I rebased to fix the merge conflict. This was just silly, I had accidentally put an empty line in states.py and this ended up getting committed.

jeanconn

Looks good to me. I think we might want a unit test of "unique" in get_starcats_as_table, but given the documented (and not documented) functional testing I think this is fine to go without it.

taldcroft · 2022-04-21T01:20:06Z

Thanks again for the reviews!

taldcroft mentioned this pull request Apr 8, 2022

Fix bug in command-not-run command set and test #222

Merged

2 tasks

taldcroft requested a review from jeanconn April 8, 2022 10:37

This comment was marked as off-topic.

Sign in to view

javierggt mentioned this pull request Apr 12, 2022

Use commands v2 for AGASC supplement update sot/agasc#135

Merged

8 tasks

sot deleted a comment from javierggt Apr 14, 2022

taldcroft requested a review from javierggt April 14, 2022 13:41

javierggt reviewed Apr 14, 2022

View reviewed changes

jeanconn reviewed Apr 20, 2022

View reviewed changes

taldcroft added 14 commits April 20, 2022 17:17

Update starcat date to MP_STARCAT command date

9d1c70a

Update tests and fix bug where params are not copied

478d0b7

Fix bug in command-not-run command set and test

b6963d5

Fix misorders between obsid cmd and NMM cmd

0f7ad17

WIP cache star catalogs

38b6628

WIP

71bf29a

Cache as a dict instead of ACATable

8ef84df

Faster caching and initial get_stars() function

4dcecb8

Improve starcats_as_table function

8412047

Change API to put start, stop first as positional args

2fd8efc

Add docs to get_starcats_as_table and more

c9214ea

Add tests and more

b5577d7

Fix misorders in a more robust way

7bebea4

Docs, config option, correct cache file path, testing

1bac78c

taldcroft added 5 commits April 20, 2022 17:20

Fix mistake in starcat caching

9ebc151

Fix missing as_dict in docstring

f8286ed

Remove set_ids arg from get_starcats

1486c8b

Better exception handling for HTTPError

087564e

Make the cache key actually unique

7b03162

taldcroft force-pushed the update-obs-parms branch from c4775e8 to 7b03162 Compare April 20, 2022 21:20

jeanconn approved these changes Apr 20, 2022

View reviewed changes

taldcroft merged commit 7664889 into master Apr 21, 2022

This was referenced Apr 21, 2022

Update sot/kadi to 5.10.0 sot/skare3#836

Closed

2022.4 sot/skare3#834

Merged

javierggt mentioned this pull request May 18, 2022

2022.5 (closed) sot/skare3#850

Closed

javierggt mentioned this pull request Jul 7, 2022

Update sot/kadi to 6.0.0 sot/skare3#879

Closed

javierggt mentioned this pull request Aug 3, 2022

2022.5 sot/skare3#899

Merged

taldcroft deleted the update-obs-parms branch September 14, 2022 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add starcat_date to observation parameters + fix caching and minor issues #221

Add starcat_date to observation parameters + fix caching and minor issues #221

taldcroft commented Apr 7, 2022 •

edited by jeanconn

Loading

taldcroft commented Apr 8, 2022 •

edited

Loading

This comment was marked as off-topic.

taldcroft commented Apr 13, 2022 •

edited

Loading

javierggt commented Apr 13, 2022 •

edited

Loading

javierggt commented Apr 13, 2022 •

edited

Loading

taldcroft commented Apr 14, 2022 •

edited

Loading

javierggt left a comment •

edited

Loading

taldcroft commented Apr 14, 2022

taldcroft commented Apr 19, 2022 •

edited

Loading

jeanconn commented Apr 20, 2022

jeanconn Apr 20, 2022

jeanconn Apr 20, 2022

jeanconn Apr 20, 2022 •

edited

Loading

taldcroft Apr 20, 2022

jeanconn Apr 20, 2022

taldcroft commented Apr 20, 2022

jeanconn left a comment

taldcroft commented Apr 21, 2022

Add starcat_date to observation parameters + fix caching and minor issues #221

Add starcat_date to observation parameters + fix caching and minor issues #221

Conversation

taldcroft commented Apr 7, 2022 • edited by jeanconn Loading

Description

Interface impacts

Testing

Unit tests

Functional tests

taldcroft commented Apr 8, 2022 • edited Loading

This comment was marked as off-topic.

taldcroft commented Apr 13, 2022 • edited Loading

javierggt commented Apr 13, 2022 • edited Loading

javierggt commented Apr 13, 2022 • edited Loading

taldcroft commented Apr 14, 2022 • edited Loading

javierggt left a comment • edited Loading

Choose a reason for hiding this comment

taldcroft commented Apr 14, 2022

taldcroft commented Apr 19, 2022 • edited Loading

jeanconn commented Apr 20, 2022

jeanconn Apr 20, 2022

Choose a reason for hiding this comment

jeanconn Apr 20, 2022

Choose a reason for hiding this comment

jeanconn Apr 20, 2022 • edited Loading

Choose a reason for hiding this comment

taldcroft Apr 20, 2022

Choose a reason for hiding this comment

jeanconn Apr 20, 2022

Choose a reason for hiding this comment

taldcroft commented Apr 20, 2022

jeanconn left a comment

Choose a reason for hiding this comment

taldcroft commented Apr 21, 2022

taldcroft commented Apr 7, 2022 •

edited by jeanconn

Loading

taldcroft commented Apr 8, 2022 •

edited

Loading

taldcroft commented Apr 13, 2022 •

edited

Loading

javierggt commented Apr 13, 2022 •

edited

Loading

javierggt commented Apr 13, 2022 •

edited

Loading

taldcroft commented Apr 14, 2022 •

edited

Loading

javierggt left a comment •

edited

Loading

taldcroft commented Apr 19, 2022 •

edited

Loading

jeanconn Apr 20, 2022 •

edited

Loading