ACLs not getting created properly when having Julie manage principles with Confluent Cloud

[shan916]

When having Julie manage principals with Confluent Cloud, the ACLs are not created properly. Also, the service account is created with format User:SA_NAME instead of just SA_NAME.

To Reproduce

1. Create a topology file with the following contents:

---
context: com.abc
lob: tech
appgroup: payment
projects: 
  - name: julie
    topics:
      - name: test-payment
        schemas:
          value.schema.file: "./schemas/example.avsc"
          value.record.type: "Payment"
        config:
          replication.factor: 3
          num.partitions: 1
        consumers: 
          - principal: "User:julie-test"
        producers:
          - principal: "User:julie-test"

2. Include the following ccloud config in the connection properties file:

julie.enable.principal.management=true
ccloud.environment=env-xxxxx
ccloud.cluster.api.key=xxxxx
ccloud.cluster.api.secret=xxxxx
ccloud.cloud.api.key=xxxxx
ccloud.cloud.api.secret=xxxxx
topology.builder.ccloud.kafka.cluster.id=lkc-xxxxx
ccloud.cluster.url=https://xxxxx:443
topology.builder.access.control.class=com.purbon.kafka.topology.roles.CCloudAclsProvider

3. Run Julie with the above topology file.

Expected behavior
The expected behavior is for the ACL to be created for the resource id associated with the test-julie account and for the service account name to be test-julie.

Screenshots
Service Account:

ACLs:

Runtime (please complete the following information):

• OS: MacOS 10.15.7
• JVM version: AdoptOpenJDK 11
• Version: 4.1.0

Additional context
If you try the ACL create operation via the Confluent Cloud REST API, the API does not create the ACL unless the numeric service account ID is specified.

The Confluent Cloud ACL API expects a principal with format principalType:principalName. If you send a request in this format, the API will return a 201 but doesn't actually create the ACL. If the numeric service account ID is specified in place of "principalName", it works.
https://docs.confluent.io/cloud/current/api.html#operation/createKafkaV3Acls

Confluent support was able to replicate the issue with the API on their end, and have escalated it internally. They also requested for us to create an issue here.

[MichaelBeausoleil]

Any timeline on when this bug will be fixed?

[shan916]

@MichaelBeausoleil we learned from a support case with Confluent Cloud that the documentation for the create ACL API is incorrect. The API only accepts the integer account ID and Confluent is working on updating the documentation https://docs.confluent.io/cloud/current/api.html#operation/createKafkaV3Acls.

I believe this is the reason Julie is not creating the ACL. It gets an inaccurate 201 response from Confluent Cloud.

@purbon ☝️

[MichaelBeausoleil]

Does that mean that I just need to change the references in the descriptor file or that there still needs to be a code change in JulieOps?

If it is the former can you please provide and example of what needs to change in the descriptor files?

If it is the later is there a timeliness for the code correction?

Thanks for the quick response.

[MichaelBeausoleil]

@shan916 can you provide a reference to the Confluent support case so I can ask Confluent about its status?

[purbon]

Hi folks, this is currently a work in progress to improve on the Confluent Cloud API. In the meantime, I will try to figure out if possible we can do something here to solve this problem in this layer. Sorry for the inconviniences, as you know this is not an official Confluent Product, and I do my best to keep it up for the latest changes and improvements.

[purbon]

Hi,
I wanted to note in the thread that I found a suitable workaround for this. I am working on a solution and will be doing a release as soon as possible, ETA end of this week beginning of next.

[MichaelBeausoleil]

Thank you for the update.

[purbon]

This should be fixed with #444

• Translation is integrated now in the API usage
• It is configurable, so whenever this bug is fixed, a user can disable it when not required anymore.
• If translation failed, JulieOps will raise an Exception.

Hope that helps!

[purbon]

Hi, quick update on this issue, you will find a release with a workaround for this issue at https://github.com/kafka-ops/julie/releases/tag/v4.1.1

hope it helps you, let me know how it goes.

[MichaelBeausoleil]

@purbon thank you very much for working on this fix. I will be trying this out today.

[purbon]

Thanks, please let me know how it goes. Remember that in the new version 4.1.1 you need to disable principal management if using the cloud, otherwise JulieOps is going to try to create your ServiceAccount as defined in the topology. You can do this by putting https://github.com/kafka-ops/julie/blob/master/example/julieops-confluent-cloud.properties#L18 as false.

…

-- Pere

On Tue, Feb 8, 2022 at 3:58 PM Michael Beausoleil ***@***.***> wrote: @purbon <https://github.com/purbon> thank you very much for working on this fix. I will be trying this out today. — Reply to this email directly, view it on GitHub <#438 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQXPBGIERBZVRJTXJTQNLU2EVQVANCNFSM5LNWN6UQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/

[MichaelBeausoleil]

I apologize for the delay on my response. The new version is working well. Thank you again for your efforts.

[purbon]

Thanks Michael, happy it is working right now for you.