Refactor/2479/simplify databackend contract

[blythed]

Description

• Only 1 type of query superduper.backends.base.query.Query
• Implementations based on execution superduper_<plugin>.Executor
• Simplifications to query API:
  • t.insert(...) instead of t.insert().execute() since we don't need to serialize insertions
  • t.update, t.delete deprecated
  • read queries have a simpler form t.filter(...).select(...).outputs(...), or t.like().(...) or t.(...).like()
  • t.get() to get one data point (eager)
  • t.ids() to get the ids (eager)
  • t.subset(ids) to subset a query
  • t.limit(n, offset=m) to get a chunk of data
• .execute() no longer returns a cursor, instead a simple list
• Remove the error prone t.column == x, replace with t['column'] == x
• Simpler serialization of "complex items" with q.dict()['documents']

[blythed]

Description

• Only 1 type of query superduper.backends.base.query.Query

• Implementations based on execution superduper_<plugin>.Executor

• Simplifications to query API:

  • t.insert(...) instead of t.insert().execute() since we don't need to serialize insertions
  • t.update, t.delete deprecated
  • read queries have a simpler form t.filter(...).select(...).outputs(...), or t.like().(...) or t.(...).like()
  • t.get() to get one data point (eager)
  • t.ids() to get the ids (eager)
  • t.subset(ids) to subset a query
  • t.limit(n, offset=m) to get a chunk of data

• .execute() no longer returns a cursor, instead a simple list

• Remove the error prone t.column == x, replace with t['column'] == x

• Simpler serialization of "complex items" with q.dict()['documents']

Still problematic:

• Inline serialization of inputs, for e.g. vector-search queries with REST API.

[jieguangzhou]

Logic that duplicates with the _add_fold_to_insert

[jieguangzhou]

We need to sort out this logic, something like this

if overwrite:
    predict_ids = ids or select.ids()
else:
    ids = ids or []
    missing_ids = select.missing_ids(predict_id)
    predict_ids = [id_ for id in ids if id_ in missing_ids]

[blythed]

I'm planning a full overhaul of this logic in an upcoming PR.

[jieguangzhou]

We only have Query and do not need MongoDBQuery, right?

[blythed]

Correct

[jieguangzhou]

We need to update our query testing utils for this generic interface.

[jieguangzhou]

Please move them to test/utils, then the ibis and mongodb can reuse it

[blythed]

I'm not sure that the query testing utility makes sense if all plugins test the same queries. TBD.