Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful handling of null typed avro fields #5723

Open
devinrsmith opened this issue Jul 5, 2024 · 0 comments
Open

Graceful handling of null typed avro fields #5723

devinrsmith opened this issue Jul 5, 2024 · 0 comments
Assignees
Labels
bug Something isn't working core Core development tasks kafka

Comments

@devinrsmith
Copy link
Member

Consider a "null" typed avro field (likely as a subfield of a record):

 {
   "name": "my_null_field",
   "type": "null",
   "default": null
 }

While it may seem strange at first, this construction may allow the caller to explicit force a digest-bump for changes that would otherwise not change the digest; for example, if documentation of a field changed, or an aliased was added or changed, it might be nice to force a digest bump. In this case, the caller could "rename" this field ("rename" in this case is really removing a field and adding a field from the perspective of avro):

 {
   "name": "my_null_field_v2",
   "type": "null",
   "default": null
 }

While the following is a stacktrace that includes an enterprise component, we should investigate how the core natively handles this situation (both consuming and producing).

io.deephaven.engine.table.TableDefinition$IncompatibleTableDefinitionException: Table definition incompatibilities: 
	Kafka table column 'my_null_field' is missing in on-disk (Kafka.BaseValuation)
	at io.deephaven.engine.table.TableDefinition.checkMutualCompatibility(TableDefinition.java:395)
	at io.deephaven.enterprise.kafkawriter.KafkaTableWriter$Registrar.register(KafkaTableWriter.java:1041)
	at io.deephaven.kafka.KafkaTools$KafkaRecordConsumerFactoryCreator.lambda$visit$3(KafkaTools.java:1169)
	at io.deephaven.kafka.ingest.KafkaIngester.lambda$new$5(KafkaIngester.java:222)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at io.deephaven.kafka.ingest.KafkaIngester.<init>(KafkaIngester.java:220)
	at io.deephaven.kafka.KafkaTools.consume(KafkaTools.java:1262)

https://avro.apache.org/docs/1.11.1/specification/_print/#transforming-into-parsing-canonical-form

@devinrsmith devinrsmith added bug Something isn't working triage kafka labels Jul 5, 2024
@devinrsmith devinrsmith added this to the 3. Triage milestone Jul 5, 2024
@devinrsmith devinrsmith self-assigned this Jul 5, 2024
@rcaudy rcaudy added core Core development tasks and removed triage labels Jul 5, 2024
@rcaudy rcaudy modified the milestones: 3. Triage, 0.36.0, 4. Unscheduled Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core Core development tasks kafka
Projects
None yet
Development

No branches or pull requests

2 participants