You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Events in a Pravega stream can be generated with or without an instance of the schema registry. An event that has been generated with a schema registry serializer will contain a header, indicating the relevant information to determine the correct schema to use. When decoding stream events, we need to know whether events have such a header or not, and decode accordingly.
Currently, we separate events into "inline" events and "regular". "inline" events are the ones that have the schema registry header.
The "inline" property is set in pravega schema registry group properties. The goal is to remove this and to auto-detect from the stream itself.
The text was updated successfully, but these errors were encountered:
We will always know the schema.
So my initial thought, and 1 possible solution, is that we could sample the first entry in a single segment. For example, split manager, or other central component, could read first event from a segment as raw bytes. We could then deserialize the bytes using the schema. If it fails, we could make an assumption that the event has schema registry header/encoding info and thus we should use schema registry serializer (i.e. "inline"). If deserialize succeeds, then we assume it is "regular".
Drawbacks are, this uses assumptions.
Events in a Pravega stream can be generated with or without an instance of the schema registry. An event that has been generated with a schema registry serializer will contain a header, indicating the relevant information to determine the correct schema to use. When decoding stream events, we need to know whether events have such a header or not, and decode accordingly.
Currently, we separate events into "inline" events and "regular". "inline" events are the ones that have the schema registry header.
The "inline" property is set in pravega schema registry group properties. The goal is to remove this and to auto-detect from the stream itself.
The text was updated successfully, but these errors were encountered: