-
-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implements 'Repository.create_blob_fromiobase'. #490
Conversation
This commit allows blobs to be constructed from implementatons of the 'RawIOBase' ABC. This allows any conformant stream implementation to be added as a blob. This is useful in the case where the contents of the blob would be too large to create from memory (as 'create_blob_fromstring' allows) and avoids having to write the data to disk (as 'create_blob_fromworkdir' allows). The latter operation is especially useful when reading data from a network socket, since it avoids having to first commit all the data to disk.
Note that this does not remove the need to have to write data to disc before writing it into the object database. That step is simply behind the |
Indeed - but it only needs to write it to disk once. Using |
I whipped this together as a proof of concept (I'm relatively new to Python/C bindings). But I had the chance to sleep on it, and perhaps it would be cleaner to expose |
Using It is not possible to write create an object without knowing the size ahead of time. Anything which does not ask for the size beforehand (or for a full buffer) must store the data somewhere else before writing to the odb. Python already has ways of making arbitrary sources look like a buffer/stream, so I'd rather use a readable stream interface rather than callbacks. |
From the discussion I understand you cannot save writing the data to disk. I have made some comments about the code, but not exhaustively, you need to thoroughly check errors and possible return values. When @carlosmn says I'd rather use a readable stream interface rather than callbacks I understand he prefers the original proposal in this PR, and not |
@jdavid I addressed the review comments. However I looked at |
Even if this does not speed up anything, it is still a nice API addition to be able to create a blob from a file like object. But the unit tests are not passing with Python 2, that needs to be fixed. |
Before I said supporting |
@jdavid thanks for the kind feedback. Looking at the link failure in the tests (and based on my extremely limited experience with Python/C interaction), is it even possible to implement it with this strategy in a way that is compatible with Python 2 and Python 3? Reading through the documentation implies this should rather be implemented at the python layer. In fact, it would be far simpler to implement a |
Okey, let make it simple:
Thanks! |
This commit allows blobs to be constructed from implementatons of the 'RawIOBase' ABC. This allows any conformant stream implementation to be added as a blob.
This is useful in the case where the contents of the blob would be too large to create from memory (as 'create_blob_fromstring' allows) and avoids having to write the data to disk (as 'create_blob_fromworkdir' allows). The latter operation is especially useful when reading data from a network socket, since it avoids having to first commit all the data to disk.
A failing test case was added which is fixed by this commit.