Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement pagination #464

Closed
Dysta opened this issue Aug 22, 2023 · 0 comments
Closed

Implement pagination #464

Dysta opened this issue Aug 22, 2023 · 0 comments

Comments

@Dysta
Copy link

Dysta commented Aug 22, 2023

Hello, we are using your modules to add a way to extract Odoo data to Airbyte with a custom connector. It works very well.

The problem is that the amount of data in our models is growing rapidly, causing each query to become slower and slower. This also causes some latency in the Odoo application when a large amount of data is extracted.

A good way to prevent slowdowns from happening when we do an extract is to paginate our extractor with Airbyte like this
image

Unfortunately, the module doesn't seem to provide a way to request just a few data in a paginated request.
The API provide a way to request the data in a paginated way with the call route. Unfortunately, this endpoint use a PATCH method that isn't supported by AirByte.
A workaround to this can be to add POST method in https://github.com/itpp-labs/sync-addons/blob/16.0/openapi/controllers/api.py#L194 and https://github.com/itpp-labs/sync-addons/blob/16.0/openapi/controllers/api.py#L173

In the screenshot above, the "page size" is the returned input number (like 100, 200, 1000, etc.) and the "start of page" is the page number (like 1, 2, etc.)

The method to pass these parameters is not just in the URL as a request, Airbyte supports a few other methods.
image

Should you be able to implement a paging system in a future release?

Thank you 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant