deep pagination not allowed on request hit at 101 page for CL API?

[purnima110895]

When I hit below url -
https://www.courtlistener.com/api/rest/v3/dockets/?date_modified__gte=2024-06-18T00%3A00%3A00Z&date_modified__lte=2024-06-18T03%3A00%3A00Z&fields=id%2Cdate_modified%2Cdate_cert_granted%2Cdate_cert_denied%2Cdate_argued%2Cdate_reargued%2Cdate_filed%2Cdate_terminated%2Cdate_last_filing%2Ccase_name_short%2Ccase_name%2Ccase_name_full%2Cdocket_number%2Cnature_of_suit%2Ccourt_id&page=101

It gives error 404-
{
"detail": "Invalid page: Deep API pagination is not allowed. Please review API documentation."
}

Any solutions for this . How can we get the data for 101 and rest of the pages?
@mlissner

[mlissner]

Did you check the documentation? It has a section on deep pagination.

[purnima110895]

@mlissner , Can you please share the link for deep pagination, I didn't find it on documentation.

[mlissner]

https://www.courtlistener.com/help/api/rest/#performance-tips

[purnima110895]

I went through this , but there is no answer for deep pagination. It only tells about choosing any field to filter the value, in my query I have taken date_modified. But then also this issue comes.

[mlissner]

Did you try narrowing the slice to be less than a full day?

[purnima110895]

Yes, it doesn't give results above page 100 , so we have to trim the datetime less than the time in the above url .
This is a big issue that the next url shown on the page doesn't return any data.

[mlissner]

Unfortunately, this is necessary to keep our servers alive. Deep pagination takes immense performance to serve.

But you can certainly slice and dice narrower than a single day and that'd fix your issue.

The other thing is we will have v4 of the API available soon, which should help: #4110