-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Confluent Cloud support #134
Conversation
@iMacTia the PR looks really good so far! Can you elaborate on the issue you were seeing with the schema classes generator? |
Oh, I was hoping my PR would run the CI and you could see it as well, is that because it's a Draft? I'll mark it ready for review 👍 But in short, it's because updating
|
Our CI needs a lot of love unfortunately - I haven't had time to go back to fix it up. :( Thanks for the note here - I'll try to look into this over the next couple of days and see if I can get some help from other team members as well. |
Thank you @dorner 🙏! |
Thanks for the quick turnaround! Ping me if I can help in any way! |
Nice work on #135, as soon as that's merged next week I'll rebase this PR and hopefully that should make tests pass. |
That was definitely the plan! It just kept getting bumped down on my list of priorities. Would be amazing if you would be willing to take a stab at that! I know that lint fails, I should raise an issue to fix the lint stuff as well. :) |
Happy to give it a try 🙌! I'll open a PR to fix both 👌 |
@iMacTia The other PR has been merged! Please fix up / resolve this one (and add an entry to the CHANGELOG under UNRELEASED). |
c9eb5a6
to
4eb3781
Compare
4eb3781
to
c4789eb
Compare
f972da2
to
3f9406d
Compare
@dorner this is now ready for review, I was able to successfully produce a message against our Confluent Cloud cluster today 🎉 ! |
Looks good! I'm going to merge and run a local test before cutting a release. |
Version has been bumped! Thanks so much for the contributions! |
Thanks for the support and the quick review 🙌 ! |
@iMacTia Would you be able to post a sanitized version of your configuration object for deimos sasl relating to Confluent Cloud? We're in a similar boat trying to get this to work and want to make sure we're setting the same things as you did in your test. |
@tjsiron Of course! I think the relevant parts are these: kafka do
seed_brokers ENV['KAFKA_BROKERS']&.split(',')
ssl do
enabled true
ca_certs_from_system true
end
sasl do
enabled true
plain_username ENV['KAFKA_SASL_USER']
plain_password ENV['KAFKA_SASL_PASS']
enforce_ssl true
end
end
schema do
backend :avro_schema_registry
use_schema_classes true
registry_url ENV['KAFKA_SCHEMA_REGISTRY_URL']
user ENV['KAFKA_SCHEMA_REGISTRY_USER']
password ENV['KAFKA_SCHEMA_REGISTRY_PASS']
end One thing that surprised my at the beginning, was the need for If you have any more specific questions (or errors to share, I have seen plenty 😂), please do let me know! |
Summary
Add support for Confluent Cloud, by adding SASL authentication.
Fixes #133
Description
This PR solves 2 authentication issues with Confluent Cloud:
The issue at the moment is that the latter change relies on a change in
avro_turf
that was introduced here, and only released as part of v1.3.0. Howeverdeimos
currently depends onavro_turf ~> 0.11
, and when I tried relaxing the dependency to allow foravro_turf 1.x
, I got an error with the schema classes generator.I managed to pinpoint the issue to a breaking change that was released with
avro_turf 1.0
. There were even plans to bring support for it since 2020 (#67), but I see no progress has been made since then.@dorner any guidance on this would be welcome. I haven't spent enough time with Deimos to feel confident fixing this compatibility issue and I'm not sure I understand your proposal about a "singleton encoder/decoder".
If this is too complicated or lengthy to implement, I might opt for temporarily monkey-patching
avro_turf
by cherry picking the basic auth change commit.Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Checklist: