alternative for fetch_page query limitation

[monikasingh20]

Hi,

In current PY3 version 'fetch_page' or 'fetch_page_async' can't be used with query that uses 'OR', '!=', or 'IN'

But it works perfectly in PY27 ndb library.

As I found in previous issue that it's a feature degradation.
So, is there any alternative for it or any way to use fetch_page with 'OR', "!=", "IN" ?
As this feature supports in python2.7, so why can't we have such feature in python3.7 ?? urgent requirement.

[andrewsg]

Thanks for your bug report. I've looked into this and apparently it's a backend issue that the Firestore/Datastore team is working on right now. As soon as that's ready we'll add support for it in the library. Thanks for your patience.

In the meantime, limit and offset should still be usable for these features.

[monikasingh20]

Thanks! for your response.
Can we expect any tentative ETA for this feature?

[andrewsg]

I didn't receive one from the Firestore team, sorry. Hopefully I'll hear more after the holiday.

[monikasingh20]

Thanks!! hope to hear from you soon.

[andrewsg]

The update I received from the Firestore team yesterday was that we've successfully implemented some of these queries on Firestore native mode, but that the Datastore mode change is pending behind some other related Datastore query work and may take significant additional time. I would strongly recommend switching to limit and offset for now.

[crwilcox]

Thanks again @monikasingh20 for raising this. We are still blocked on implementation of In/Not-in queries for Cloud Datastore. This incompatibility with the Python 2 version is called out in documentation to hopefully help users work around this. https://googleapis.dev/python/python-ndb/latest/query.html#google.cloud.ndb.query.Query.fetch_page

This feature is being added to Cloud Datastore however I don't have a timeline for when it will be released at this time.

[punitaojha]

Hello @crwilcox, do we have any latest updates on the same. I was waiting if I could use it.

[crwilcox]

I still don't have a firm timeline, though it is likely sometime next year.

[denisgrenader]

HI @crwilcox -- any news on this front?

[kkroening]

What seems to make this challenging is that ndb currently has to run multiple queries in parallel in order to support IN/OR/!= queries, and the _MultiQueryIteratorImpl doesn't currently know how to deal with the fact that it's managing multiple sub-query cursors. I suspect that there could be a way to work around this by having _MultiQueryIteratorImpl.{cursor_before,cursor_after} simply combine the cursors by concatenating all the sub-query cursor values, and then having _MultiQueryIteratorImpl.__init__ do the reverse by unpacking the start_cursor.

From an external perspective, the _MultiQueryIteratorImpl would look just like any other QueryIterator implementation, accepting a seemingly-ordinary start_cursor and producing seemingly-ordinary cursor_before() and cursor_end() values, but in fact, these cursors would contain all of the sub-query cursors essentially multiplexed together.

Just an idea; not sure if there's any reason that this wouldn't work. Of course, if Datastore/Firestore itself eventually natively supports IN/OR/!=, then this becomes moot, but it could be a while before that happens and ndb gets updated to use it, and the datastore emulator is updated as well - so perhaps if someone's bored, it might be worth trying this out, at least as a science experiment to see if it's a viable approach.

[jacobg]

@crwilcox Is support for this type of query still in the works? Thanks.

[neurino]

Hello, any update on this topic? This is very limiting: using IN of course returns more results and thus needs pagination :(

[meredithslota]

Support for IN/NOT_IN/NOT_EQUAL operators was released in Cloud Datastore in https://github.com/googleapis/python-datastore/releases/tag/v2.7.0, if that's helpful.

[pacssee]

Still getting this error with google-cloud-datastore==2.7.0:
TypeError: Can't use 'fetch_page' or 'fetch_page_async' with query that uses 'OR', '!=', or 'IN'.

Also, google-cloud-ndb 1.11.1 (current version) has requirement google-cloud-datastore<2.0.0dev,>=1.7.0 which makes it incompatible with 2.7.0

[rwhogg]

We recently upgraded our dependencies, so now we depend on at least google-cloud-datastore 2.7.2.

[michaelkariv]

What is the latest on this? I rely on IN for my Py2 GAE code, have to migrate because of the py2 GAE end of life early next year.
UPDATE 2023-04-04, I have updated google-cloud-ndb to the currently latest 2.1.1, and it updated google-cloud-datastore to the currently latest (per pip) 2.15.1. Still get the error.

[JohnGale87]

Please can we have an update on this issue?

[JohnGale87]

Still looking for an update on this issue, any news?

[sorced-jim]

Not complete fixes, but #951 & #950 allow supporting != & IN (within the server limits) for fetch_page.

[tlatin]

@sorced-jim what size is the server limits?

Looks like this went out with the 2.3.0 python-ndb release.

[sorced-jim]

The server side limits are documented at https://cloud.google.com/datastore/docs/concepts/queries#in. It's currently 30.