Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem creating entire starcats.db cache file in one session #252

Closed
taldcroft opened this issue Sep 14, 2022 · 1 comment
Closed

Problem creating entire starcats.db cache file in one session #252

taldcroft opened this issue Sep 14, 2022 · 1 comment

Comments

@taldcroft
Copy link
Member

As noted in sot/agasc#135:

In [1]: from agasc.supplement.magnitudes import star_obs_catalogs
   ...: 

In [2]: stars_obs_v2 = star_obs_catalogs.get_star_observations()
   ...: 

HASH: Out of overflow pages.  Increase page size
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-2-b3fe61c90950> in <module>
----> 1 stars_obs_v2 = star_obs_catalogs.get_star_observations()

~/git/agasc/agasc/supplement/magnitudes/star_obs_catalogs.py in get_star_observations(start, stop, obsid)
     23 
     24     with commands.conf.set_temp('commands_version', '2'):
---> 25         catalogs = commands.get_starcats(start=start, stop=stop, obsid=obsid)
     26         observations = Table(commands.get_observations(start=start, stop=stop, obsid=obsid))
     27         for cat in catalogs:

~/git/kadi/kadi/commands/observations.py in get_starcats(start, stop, obsid, scenario, cmds, as_dict)
    365                     starcat = starcat_dict
    366 
--> 367                 starcats_db[db_key] = starcat_dict
    368 
    369             starcats.append(starcat)

~/miniconda3/envs/ska3/lib/python3.8/shelve.py in __setitem__(self, key, value)
    123         p = Pickler(f, self._protocol)
    124         p.dump(value)
--> 125         self.dict[key.encode(self.keyencoding)] = f.getvalue()
    126 
    127     def __delitem__(self, key):

error: cannot add item to database

I spent a good while doing a deep dive into this problem, which is slow because it only shows up after a couple minutes of churning.

  • It has nothing to do with memory per se. I was able to replicate the problem with the activity profiler going and the process was never taking more than 1.2 Gb before crashing.
  • Googling the error produces a number of hits but I couldn't find anything useful. Maybe I didn't click the right link or spend enough time digging into the result pages.
  • If you break the shelf creation process in into smaller parts (say 5 years at a time) and exit Python each time, then I haven't see a problem..
  • This would suggest that breaking a large query (e.g. get_starcats()) into bits where the shelve file is closed and re-opened might help. MADDENINGLY, this didn't work and I got the same error at the same place. So it looks like there is some kind of leak in shelve that causes problems even when the shelf file is closed. Go figure.

So this is in a somewhat unhappy place. The workaround is incrementally building up the starcats.db cache file in 5 year chunks. Eventually I'll do something better.

@taldcroft
Copy link
Member Author

Closed by #253. I confirm that getting every star catalog in the mission starting from scratch (no starcats_cache.db file) does not crash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant