Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support partial extents request #83

Open
ahadas opened this issue Jun 13, 2022 · 0 comments
Open

Support partial extents request #83

ahadas opened this issue Jun 13, 2022 · 0 comments
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance

Comments

@ahadas
Copy link
Member

ahadas commented Jun 13, 2022

Currently we support getting image extents for an entire image. With large
images on very fragmented file system, this can be very slow, delaying
backup, download disk, or upload disk (we use extents to determine the size).

With partial extents, we can get extents for part of the image, e.g. 1g, and
start downloading data for the received extents while getting the extents
for the next segment. This is the way other backends provide extents to
callers.

For example when you iterate over nbd backend extents:

for extents in backend.extents():
    ...

The backend does not get all the image extents at once. It get extents for
the first 2g, and start yielding them. When all extents are consumed, the
backend fetch extents for the next segment, until the end of the image
is reached.

From the client point of view this partial extent retrieval is transparent
and delays with large images and fragmented file systems are minimized.

We need to provide the same mechanism for the http backend. To implement this
we must have a way to specify ranges when getting extents.

Provide start= and length= query parameters to get partial extents.

Example requests:

# Get all extents from offset 0. length is (size - offset) if not
# specified.
GET /images/ticket-id/extents?context=zero&start=0

# Get extents from offset 0 to 1073741824
GET /images/ticket-id/extents?context=zero&start=0&length=1073741824

# Same, offset is 0 if not specified.
GET /images/ticket-id/extents?context=zero&length=1073741824

# Get extents from offset 52613349376 up to 53687091200
GET /images/ticket-id/extents?context=zero&offset=52613349376&length=1073741824

On the server side, pass the arguments to the backend without any
modification.

On the http backend, change extent() to get 1G extents per call
from the server, and get the next extents when needed.

Currently we cache extents in the client. I'm not sure how caching
should be implemented. We can remove the caching since no other
backend caches extents. The main reason for caching was not having
a good way to get image size, so we implemented size() using extents
request.
for partial extents.

Original bug: https://bugzilla.redhat.com/1924940

@nirs nirs added enhancement Enhancing the system by adding new feature or improving performance or reliability performance labels Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance
Projects
None yet
Development

No branches or pull requests

2 participants