'Connection.lookup' can return 'missing' and 'deferred' results

[tseaver]

Do we need to support those fields?

[tseaver]

Options:

• Have the user pass optional missing and / or deferred lists into lookup(), and populate them IFF they are not None.
• Change lookup() to always return a three-tuple, (found, missing, deferred), where each item is a list (of entities, key-only entities, and keys, respectively).

[dhermes]

I prefer the first option or just a different verb like fetch() versus fetch_page() with ndb.

[silvolu]

IRT "do we need to support", they come quite cheap in the end, so it'd be good to have them.
@pcostell, out of curiosity, how likely is that deferred contains results?
I don't have any strong opinions about your proposals.

[pcostell]

A small percentage of responses have deferred results. The field will only be populated if there were too many keys requested or the result size was too large. I would consider reissuing a lookup to get any deferred keys under the hood so that the user just gets all the results they requested.

[silvolu]

I like @pcostell proposal.

[dhermes]

I do too, with the caveat that we should let people pass in a boolean to disable the "more RPCs without telling the user" behavior and under that flag we could return the keys that were deferred?

[tseaver]

Can we have the value passed in to disable extra under-the-covers RPCs be a list, into which the contents of the deferred field from the response would be copied? We could handle missing similarly. E.g.:

deferred = []
missing = []
entities = con.lookup(dataset_id, key_pbs, deferred=deferred, missing=missing)

If deferred isn't a list, then we do the "try again" dance as suggested by @pcostell. If missing isn't a list, then we just ignore it (the status quo).

[silvolu]

SGTM. @dhermes ?

[dhermes]

Yes I like this approach. Simple and convenient.

[tseaver]

@pcostell wrote:

I would consider reissuing a lookup to get any deferred keys under the hood so that the user
just gets all the results they requested.

Should the code sleep for some interval before retrying the deferred keys?

[pcostell]

No need to sleep. There were just too many results in that request.

[dhermes]

I feel like we should do something if a user requests 100,000 entities in a fetch other than just fetching until we have them all.

We already have reports of people trying to fetch large numbers of entities without proper understanding of what the implications are. We can provide guidance by throwing exceptions or printing error / warning messages. At the very least, people will wonder what is taking so long. (Even if it is PEBKAC)