Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to tell whether Pravega events have been generated with schema registry #20

Open
adrdc opened this issue Mar 16, 2021 · 1 comment · May be fixed by #42
Open

Ability to tell whether Pravega events have been generated with schema registry #20

adrdc opened this issue Mar 16, 2021 · 1 comment · May be fixed by #42
Assignees

Comments

@adrdc
Copy link
Contributor

adrdc commented Mar 16, 2021

Events in a Pravega stream can be generated with or without an instance of the schema registry. An event that has been generated with a schema registry serializer will contain a header, indicating the relevant information to determine the correct schema to use. When decoding stream events, we need to know whether events have such a header or not, and decode accordingly.

Currently, we separate events into "inline" events and "regular". "inline" events are the ones that have the schema registry header.
The "inline" property is set in pravega schema registry group properties. The goal is to remove this and to auto-detect from the stream itself.

@adrdc
Copy link
Contributor Author

adrdc commented Mar 24, 2021

We will always know the schema.
So my initial thought, and 1 possible solution, is that we could sample the first entry in a single segment. For example, split manager, or other central component, could read first event from a segment as raw bytes. We could then deserialize the bytes using the schema. If it fails, we could make an assumption that the event has schema registry header/encoding info and thus we should use schema registry serializer (i.e. "inline"). If deserialize succeeds, then we assume it is "regular".
Drawbacks are, this uses assumptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant