Add support for TIBCO's Avro serializer/deserializer #558
Conversation
…and the TIBCO Avro client jar is on the classpath, then the TIBCO Avro wire format will be used for serialization/deserialization of Avro messages. Also cleaned up a few small unrelated code typos.
|
Hi there - just checking in to see if anyone's had a chance to take a look at this PR. Let me know what you think - what you'd like changed if anything, etc. Thanks! |
| @@ -31,6 +31,7 @@ public Connection(@Parameter String name) { | |||
| String url; | |||
| String basicAuthUsername; | |||
| String basicAuthPassword; | |||
| String type; | |||
There was a problem hiding this comment.
Change this one to an Enum please and put a default one to Confluent
| config.putAll(this.kafkaModule.getConnection(clusterId).getProperties()); | ||
| deserializer.configure(config, false); | ||
| } else { | ||
| deserializer = (Deserializer) Class.forName("io.confluent.kafka.serializers.KafkaAvroDeserializer").getDeclaredConstructor(SchemaRegistryClient.class).newInstance(this.kafkaModule.getRegistryClient(clusterId)); |
There was a problem hiding this comment.
Why use of reflection here ?
The class is in class path
|
sorry @ebowden-tibco , completely forgot. Also you need to provide :
And it will be fine 👍 |
|
Thanks for the feedback! I'll update the PR soon with the requested changes. |
…ng how to configure and use the TIBCO schema registry. Changed schema registry 'type' option to an Enum and simplified the code a bit. Added support for the CLASSPATH environment variable to the Docker image. Fixed a few nearby doc typos.
|
Perfect !! |
|
@tchiotludo, out of curiosity, could I modify the build.gradle to add a new Maven repo (company Nexus) Line 33 in aada01e with the TIBCO Jar in it, and add the reference to the TIBCO jar Line 107 in aada01e I will still need to make the changes for the schema repository type in the config file. |
|
I don't think so, the license will not allow to embed a proprietary jar on this opensource project. |
|
Of course not here, but if I clone it locally and make the changes. |
|
Hi @ebowden-tibco , |
|
@xpr3sso - You shouldn't need any config changes. Could you give an example config snippet that fails for you? Do you just not have a "schema-registry" section in your config at all? Does everything work immediately before commit aada01e ? ie, if you checkout at commit 477c704, do things work again? I'm not sure how these changes would've affected AKHQ use without a registry, but I'm new to the AKHQ code base, so anything's possible. |
|
@ebowden-tibco just checked 477c704 and this is working fine for me :) Are you able to reproduce this? My connections section looks like: connections:
localhost:
properties:
bootstrap.servers: "localhost:9092" |
|
Doh! Yes, I can reproduce it here. I did all my testing with a schema-registry section in the config file. Tested with and without the new schema registry "type" option present, but didn't think to test with/without the entire schema-registry config block itself. I'll try to get this fixed today. |
This PR works as-is for me for deserializing & displaying messages that were serialized by the TIBCO Avro serialization library. Since the TIBCO client library isn't currently available via a public Maven repository, it must be manually downloaded (as part of the "TIBCO® Messaging - Schema Repository for Apache Kafka - Community Edition" package at https://www.tibco.com/products/tibco-messaging/downloads) and then AKHQ must be started via setting the classpath with the TIBCO jar and its dependencies specified in addition to AKHQ's jar, rather than just using the -jar option. Meaning - with this PR, instead of starting AKHQ like:
You'll need to start it by calling:
Additionally, in your AKHQ config file's 'schema-registry' section, you'll need to add a new 'type' option set to 'tibco' if you're using a TIBCO schema repository with the TIBCO serializer/deserializer. E.g.:
(Also note the URL; it should be set to the URL of the TIBCO FTL Server used by the TIBCO schema repository with "/schema/v1" appended at the end as shown above.)
This PR contains just a minimal number of changes to allow full compatibility with the TIBCO schema repository and Avro serializer client library. It may be worth discussing more substantial changes to AKHQ to allow for the flexibility to specify a SerDe class and options to use on a per-topic-regex basis (similar to what's currently used for the JSON deserializer) in the future, which would allow AKHQ to support any serialization format out there that has an existing Kafka SerDe for it without needing to add explicit support for the format to AKHQ each time.