-
-
Notifications
You must be signed in to change notification settings - Fork 278
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache attributes #220
Cache attributes #220
Conversation
zarr/attrs.py
Outdated
self.synchronizer = synchronizer | ||
|
||
def _get(self): | ||
if self.key in self.store: | ||
d = json.loads(text_type(self.store[self.key], 'ascii')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a general note, in the future for remote datasets we might want to minimize multiple touches of self.store
. I can imagine wanting to change things like this to code like the following instead:
try:
data = self.store[self.key[
except KeyError:
d = {}
else:
d = json.loads(text_type(data))
This is just a general comment though, not something that should necessarily be done here. I'll bring it up again if we find this a challenge when profiling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good suggestion, asking for permission is a hard habit to break :-)
@alimanfoo - is it possible or could it be possible to load all the attributes of an array/store at once? Caching is great but accessing each attribute individually will still be pretty costly. |
You can get all the attributes of an array or group in one call with
o.attrs.asdict(). Btw all attributes for a single array or group are stored
together in a single JSON resource, so with this caching PR all attributes
of an array or group will be cached as soon as the first one is accessed,
so performance should be same as manually calling asdict(). Hope that makes
sense.
On Tue, 26 Dec 2017 at 20:55, Joe Hamman ***@***.***> wrote:
@alimanfoo <https://github.com/alimanfoo> - is it possible or could it be
possible to load all the attributes of an array/store at once? Caching is
great but accessing each attribute individually will still be pretty costly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/alimanfoo/zarr/pull/220#issuecomment-354012284>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAq8Qu76IULssdeTZuu1CNZjdVH2bICLks5tEV1BgaJpZM4RLy6r>
.
--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health <http://cggh.org>
Big Data Institute Building
Old Road Campus
Roosevelt Drive
Oxford
OX3 7LF
United Kingdom
Phone: +44 (0)1865 743596
Email: alimanfoo@googlemail.com
Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/
Twitter: https://twitter.com/alimanfoo
|
Is there a way to toggle caching on an existing object? Also is there a way to clear the cache? |
Btw I've also now exposed a |
…o cache-attrs-20171224b
@jakirkham you can turn caching off with |
Thanks @alimanfoo. That sounds great. |
This PR adds support for caching user attributes, enabled by default. Resolves #218.
TODO: