Asynchronous Writes/Reads

[aosterthun]

This feature is currently only available in c++. Any chance somebody could write the python wrapper for this in the near future ?

If not, dispatching the write operation in a new thread should work fine, shoudn't it ?

[jakebolewski]

All the IO / cpu intensive operations do give up the GIL, so usually the best option is to dispatch the write on another thread and then get a future to that (blocking) operation.

Full async support (integration with the event loop), probably should follow after making the above operation (which you can do now) a bit more user friendly and documented.

[ocehugo]

Full async support (integration with the event loop), probably should follow after making the above operation (which you can do now) a bit more user friendly and documented.

Hi @jakebolewski - is this too far away still? I'm happy to code some manual stuff but some documentation with any idiosyncrasies around the c++/python api would help.

PS: I'm available to help since I got an ongoing related project that would benefit from this.

[ihnorton]

Hi @ocehugo, are you looking specifically for integration with Python 3's async/await? At some point I looked in to this and saw that Cython doesn't support the async annotation on cdef functions, but I'll take a look again soon to try to work around that (it's not clear to me how strict the interpreter is about functions defined in C -- at worst we can probably hack in the attribute it wants).

Pull-requests very welcome of course! 😄

[ocehugo]

@ihnorton yeap - python 3.6+ probably, since some breaking API changes happened before that.

I'm happy to push something but I'm not familiar with the API internals and the C++ code. I'm just a very ordinary user ATM. Im a bit rust in cython but played with it before. If you could get me some direction or where to look for the core things It can help a lot.

FYI: My main objective is to do Async fetching on object storage, mostly dense gridded stuff and associated metadata with time-x-y ranges.