Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for objects that support buffer protocol for "read" calls #62

Merged
merged 1 commit into from
Jan 29, 2024
Merged

Add support for objects that support buffer protocol for "read" calls #62

merged 1 commit into from
Jan 29, 2024

Conversation

ericpruitt
Copy link
Contributor

No description provided.

@ericpruitt
Copy link
Contributor Author

This adds support for the buffer protocol when executing the read I/O callback. This change does not make reads zero-copy. I haven't dug into the FUSE code enough to understand if zero-copy between Python and the underlying C library is even possible, so I would be interested in hearing from the maintainers on this and, if it is possible, what exactly should be changed.

@sdelafond
Copy link
Collaborator

This change does not make reads zero-copy. I haven't dug into the FUSE code enough to understand if zero-copy between Python and the underlying C library is even possible, so I would be interested in hearing from the maintainers on this and, if it is possible, what exactly should be changed.

I'm not sure if it's possible either, so I won't insist on it. I'll review your change and merge it later this week after I run some tests on my end.

@ericpruitt
Copy link
Contributor Author

The operations struct documentation says the memory buffer will be freed by the caller for "read_buf." The documentation for "read" doesn't explicitly say that, but I don't see anything that suggests a different behavior is possible i.e. a flag to indicate that the buffer should not freed.

@sdelafond sdelafond merged commit 6c2732f into libfuse:master Jan 29, 2024
8 checks passed
@glensc
Copy link
Contributor

glensc commented Feb 16, 2024

@ericpruitt please share python code that would correspond to the changes here.

@ericpruitt
Copy link
Contributor Author

@glensc , I don't understand what you're asking for. Adding support for the buffer protocol didn't require any changes to the pure Python portion of the library.

@sdelafond
Copy link
Collaborator

I'm guessing he was asking for an example of client code using the buffer protocol.

@ericpruitt
Copy link
Contributor Author

ericpruitt commented Feb 16, 2024

This requires literally no changes to existing code because Python bytes, the data type that you would already have to use, support the buffer protocol without any changes. This just reduces the amount of churn by CPython. Normally, doing something like data[...:...] to get the some chunk of data from an existing block in-memory will result in Python duplicating some of the data. You should be able to see this if you try the following code on a *nix system and monitor its memory consumption:

>>> x = bytes(b'\0' * 10000000000)
>>> y = x[:10000000000-10]

What the C change does is effectively makes the underlying code work like this:

>>> x = bytes(b'\0' * 10000000000)
>>> y = memoryview(x)

At which point running z = y[:10000000000-10] should only show a negligible increase in memory. So in short, the code is similar to casting all inputs with memoryview, but it doesn't require the user to do anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants