Update Exported types to include custom objects (eg. _deephaven_object on a python object)

[mofojed]

Allow exporting a custom type

• Support exporting a custom payload (base64 encoded string? opaque format?)
• Allow passing table tickets (and input tables as well for possible two way communication). Tables can be fetched in another pass.
• Support "complex" custom type information, without sending the full data payload (metadata)

Related to discussion: #1468
#1468 (comment)

[niloc132]

Summary of call yesterday: There are three basic cases where generic serialized objects will be made available, table attributes, console variables, and application fields.

• As before, console and application use cases will allow the server to notify the client when a new object exists, with its ticket, name, type, and some human-readable fields for name and description.
• A generic fetch grpc call should exist that can handle these tickets (which probably should take the expected type as well). The server will respond with a message made up of some serialized bytes, and any other tickets that are embedded in that payload. It is then assumed that the client DH API will offer the ticket instances to the client user, along with the payload itself.
• In theory a table can be sent in this way, with the ExportedTableCreationResponse pb sent as the bytes, but in practice it would make more sense to continue to use TableService or otherwise directly interact with FlightService to address the data.
• Table attributes (and other future cases that may exist) may refer to these arbitrary objects too, though this will not mean that more data will be sent to the client, but instead that the client would only get the attribute keys and types, and will have a way to fetch the unsent attribute content it needs from a getTableAttribute method.

Proposed rough grpc changes:

• application.proto replaces FieldInfo.field_type's FieldType with a string
• console.proto should use FieldInfo instead of VariableDefinition
• Introduce a message for SerializedObject with bytes data and repeated Ticket tickets or the like
• Some service should support a fetchObject grpc method which takes a ticket (and a type string?) and returns a SerializedObject
• Some service should support a fetchTableAttribute grpc method which takes a table ticket (and a type string?) and a attribute key, and returns a SerializedObject. This implies/requires that it is possible to fetch a Table as a SerializedObject, such as for trees.
• (optional/eventual) Figures can be sent as a protobuf SerializedObject wrapping the figure descriptor, rather than supporting its own custom method to fetch - same will be true for pandas tables.

Internal Java changes:

• Service loader/DI addition to offer DeephavenSerializer instances, which can claim objects that they can send, declare a type name that they are responsible for, and write those instances to SerializedObject instances.
• Figure serialization should be embedded in this serializer mechanism, possibly other python objects (even tables?) should be handled (i.e. "unwrapped") in this way too.

Flight Schema changes:

• unused attributes should send their type string, rather than an empty string value
• (optional) all sent attributes should be encoded using SerilaizedObject instead of plain string. This would suggest that SerializedObject come with a string type field, so that the types are apparent to the consumer. The reason to add this feature would be to allow sending things like tree info in a single attribute instead of several nested psuedo-attrributes - and if we're formalizing a generic way to send data rather than simple conventions, then we should likewise formalize what type an arbitrary attribute has.

JS API changes:

• JsTable.getAttribute will be able to return a generic Promise rather than only table
• IdeConnection.getObject will be able to return a custom object of some kind, wrapping the fields of SerializedObject

[mofojed]

@niloc132 / @nbauernfeind One question I forgot to raise is how will a Python extension hook into this to advertise it's custom types? Is it adding a custom attribute onto the PyObject that indicates it should be an object that's exported (similar to how we do with IsWidget.GET_TABLE_ATTRIBUTE, or like in my PoC with https://github.com/mofojed/deephaven-core/blob/42e724cbd50ac5d58256f72b9ca80673e388ea89/DB/src/main/java/io/deephaven/db/util/IsWidget.java#L15)? Or some other way?

[nbauernfeind]

@mofojed I like the getDeephavenObject approach as it is flexible for python users. However, it probably needs to return an ApplicationField wrapping the serialized string. We should support both generic serialization of type T (the service loader idea in Colin's comment), but we'll need to be flexible if the user wants to (or needs to) serialize an immutable object themselves.

[rcaudy]

Superseded by #1766