feat: Proper working cursor

[another-rex]

The current paging/cursor implementation is slightly hacky and fails at various edge cases, and for many query types does not work at all, e.g. (queries without ecosystem, semver queries...).

When initially implemented, the assumption was almost all API calls resulted in one datastore query. This is no longer the case.

This refactor/rework adds:

• New cursor type
  • which can represent 3 states (Start, End, InProgress) in a type safe way (through enums, thanks for the ideas @michaelkedar @DonggeLiu!)
  • Also stores the ndb.Cursor inside
  • Also stores which query number the ndb.Cursor is for when there are multiple queries
  • can serialise to a page token, and deserialise in a backward compatible fashion.
• Reworked logic to centralise cursor handling to make the code more readable and predicable:
  • Cursor state is stored in context, rather than passed through a tuple through return types
  • Query skips are determined next to each query.iter() function, where the datastore queries actually happens, rather than in outer functions.
• Context now accounts for the number of datastore queries that has been performed
  • This is used to know which query a cursor belongs to
• Minor refactors to query_generic_helper() to remove unnecessary arguments being passed in
• Add a skipIf on env variable to allow actually running the pagination tests locally.
• Modify the tests to run faster (by lowering the sleep duration haha),
• Modify tests to take in unittest arguments of which tests to run from the second argument onwards (rather than having the gcloud service account always being the last argument)

[oliverchang]

nice!

[andrewpollock]

This is awesome

[oliverchang]

When is this ever true, given https://github.com/google/osv.dev/pull/2514/files#r1730697012 ?

[another-rex]

Ah that should be 1

[another-rex]

Actually removed this entirely, there is no reason to not return this as part of the page token.

[oliverchang]

LGTM with some minor remaining comments. Awesome stuff!

[michaelkedar]

nit: Since it's the only thing that uses it , this could possibly go inside of QueryCursor:

class QueryCursor:
  class _State(Enum):
    ENDED = 0
    # ...
    
  _cursor_state: _State = _State.ENDED

  @property
  def ended(self) -> bool:
    return self._cursor_state == QueryCursor._State.ENDED

That said, I don't think I have a preference either way.

[another-rex]

I think it's just a bit clearer to be private but at the top level to avoid having to write the class name repeatedly.