Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SASL connection to Eventhubs using kafka fails 1.38.0 onwards #2470

Closed
DeepanshuA opened this issue May 5, 2023 · 4 comments
Closed

SASL connection to Eventhubs using kafka fails 1.38.0 onwards #2470

DeepanshuA opened this issue May 5, 2023 · 4 comments

Comments

@DeepanshuA
Copy link

DeepanshuA commented May 5, 2023 •
edited
Loading

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

| Sarama | Kafka | Go |
|1.38.0|2.0.0|1.20.1|

Configuration

What configuration values are you using for Sarama and Kafka?



Logs


When filing an issue please provide logs from Sarama and Kafka if at all

possible. You can set sarama.Logger to a log.Logger to capture Sarama debug

output.


logs: CLICK ME



time="2023-05-05T20:13:49.447569+05:30" level=debug msg="client/metadata fetching metadata for all topics from broker sasl-issue.servicebus.windows.net:9093\n" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.4581451+05:30" level=debug msg="Completed pre-auth SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER]" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.6997856+05:30" level=debug msg="SASL authentication successful with broker sasl-issue.servicebus.windows.net:9093:4 - [0 0 0 0]\n" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.6999026+05:30" level=debug msg="Connected to broker at sasl-issue.servicebus.windows.net:9093 (unregistered)\n" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.9402314+05:30" level=debug msg="client/metadata got error from broker -1 while fetching metadata: read tcp 172.21.220.249:41466->13.66.149.8:9093: read: connection reset by peer\n" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.9406636+05:30" level=debug msg="Error while closing connection to broker sasl-issue.servicebus.windows.net:9093: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 172.21.220.249:41466->13.66.149.8:9093: write: broken pipe\n" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.9407381+05:30" level=debug msg="client/metadata no available broker to send metadata request to" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.9407646+05:30" level=debug msg="client/brokers resurrecting 1 dead seed brokers" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.9408188+05:30" level=debug msg="Closing Client" app_id=orderapp component="kafkabind (bindings.kafka/v1)" instance=LAPTOP-J62MA7E9 scope=dapr.contrib type=log ver=unknown
time="2023-05-05T20:13:50.9409944+05:30" level=warning msg="Error processing component, daprd process will exit gracefully" app_id=orderapp instance=LAPTOP-J62MA7E9 scope=dapr.runtime type=log ver=unknown
time="2023-05-05T20:13:50.9410541+05:30" level=fatal msg="process component kafkabind error: [INIT_COMPONENT_FAILURE]: initialization error occurred for kafkabind (bindings.kafka/v1): kafka: client has run out of available brokers to talk to: read tcp 172.21.220.249:41466->13.66.149.8:9093: read: connection reset by peer" app_id=orderapp instance=LAPTOP-J62MA7E9 scope=dapr.runtime type=log ver=unknown







Problem Description


Dapr started to use 1.38.0 and then 1.38.1 sarama version.

When trying to connect to Event hubs using kafka via Dapr, this issue occurs.


If sarama 1.37.2 version is used, connection is successful and no issue occurs.


Tried to debug as well, if some new property needs to be set for SASL connection after 1.38.0 onwards, but couldn't find anything relevant.


It fails at this place: https://github.com/Shopify/sarama/blob/9127f1c432c08b9a766fdb0931bfd0da80dddde0/broker.go#L1133


Related to dapr/components-contrib#2755

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
          

  

    
  


  

      

    @DeepanshuA
DeepanshuA



    mentioned this issue
    
      May 5, 2023
    





  


  

      
  
  Closed

  







  









      

  
      



  

  @dnwe





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

      

      
            dnwe
  

      

      

      commented


        May 16, 2023


      
    


  





      


        


  
    

      
          
@DeepanshuA hmm, the only change that looks relevant is #2388 when we started sending MetadataRequest V7 if you have Version in your sarama.Config as V2_1_0_0 or newer. It's possible that EventHubs isn't supporting this. Can you try setting your sarmaa.Config Version down to V2_0_0_0 and test?

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
          

  

    
  


  

      

    @616b2f
616b2f



    mentioned this issue
    
      Jun 8, 2023
    





  


  

      
  
  Closed

  







  









      

  
      



  

  @github-actions





  

    

      
  
  

    
      

        

              This comment was marked as outdated.

        

        

          Sign in to view
        

      

    
  



    













      

  
            

    

      
    

    


      

          @github-actions
github-actions
bot



          added
  the 

  stale

  Issues and pull requests without any recent activity
 label


      Aug 14, 2023

    

  








      

  
      



  

  @dnwe





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

      

      
            dnwe
  

      

      

      commented


        Aug 15, 2023


      
    


  





      


        


  
    

      
          
The problem with supporting Azure Event Hubs is that it isn't Apache Kafka under the covers, it's an intermediate proxy that supports a subset of the Kafka APIs at various versions and then maps them onto Event Hubs protocol(s) at the backend. As as result Sarama's current mechanism of specifying the KAFKA_VERSION to determine what protocol versions to support and use doesn't really work properly with Event Hubs because it supports an unusual set of protocols and even defines minimum versions for ProduceRequest (v3) and FetchRequest (v4). For some reason EventHubs is very behind on FetchRequest and MetadataRequest so the max configuration you can use in Sarama is V1_0_0_0.


Here are the equivalent max versions for Kafka 1.0 and EventHubs:


    APIs                                 1.0  EventHubs
0   ProduceRequest                       v5   v7
1   FetchRequest                         v6   v6
2   ListOffsetsRequest                   v2   v7
3   MetadataRequest                      v5   v5
8   OffsetCommitRequest                  v3   v8
9   OffsetFetchRequest                   v3   v8
10  FindCoordinatorRequest               v1   v1
11  JoinGroupRequest                     v2   v7
12  HeartbeatRequest                     v1   v4
13  LeaveGroupRequest                    v1   v4
14  SyncGroupRequest                     v1   v5
15  DescribeGroupsRequest                v1   v5
16  ListGroupsRequest                    v1   v4
17  SaslHandshakeRequest                 v1   v1
18  ApiVersionsRequest                   v1   v3
19  CreateTopicsRequest                  v2   v6
20  DeleteTopicsRequest                  v1   v2
22  InitProducerIdRequest                v0   v1
23  OffsetForLeaderEpochRequest          v0   v0
32  DescribeConfigsRequest               v0   v2
36  SaslAuthenticateRequest              v0   v1
37  CreatePartitionsRequest              v0   v1
42  DeleteGroupsRequest                       v0
47  OffsetDeleteRequest                       v0


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            

    

      
    

    


      

          @github-actions
github-actions
bot



          removed
  the 

  stale

  Issues and pull requests without any recent activity
 label


      Aug 15, 2023

    

  



  

  

    
  


  

      

    @dnwe
dnwe



    mentioned this issue
    
      Aug 29, 2023
    





  


  

      
   Merged

  





  
  

      
        
      
      

  
    
    
  


    3 tasks
  





  




  


  
    
  
  

    
  dnwe 

      added a commit
      that referenced
      this issue

    
      Aug 30, 2023
    
    

      


  

    

      


  

      
        @dnwe
  





      

        
          fix(client): force Event Hubs to use V1_0_0_0
        

          
            
          

      


      

          

    
    
    

  

      


      

          

    
    
    

  

  

    
  



      


      
      

        
          f1baac8
        
      

    

  

    

      
The problem with supporting Azure Event Hubs is that it isn't Apache
Kafka under the covers, it's an intermediate proxy that supports a
subset of the Kafka APIs at various versions and then maps them onto
Event Hubs protocol(s) at the backend. As as result Sarama's current
mechanism of specifying the KAFKA_VERSION to determine what protocol
versions to support and use doesn't really work properly with Event Hubs
because it supports an unusual set of protocols and even defines minimum
versions for ProduceRequest (v3) and FetchRequest (v4). For some reason
EventHubs is very behind on FetchRequest and MetadataRequest so the max
configuration you can use in Sarama is V1_0_0_0 and the minimum is
V0_11_0_0.

As we have recently bumped the default Version to V2_1_0_0 and support a
wider range of protocol versions, we're more likely to see issues raised
by Event Hubs users unless they have already pinned their version
correctly. To try and prevent this, attempt to detect use with Event
Hubs by inspecting the bootstrap broker addresses and overriding the
Version.

Contributes-to: #2470

Signed-off-by: Dominic Evans <dominic.evans@uk.ibm.com>

    




    

  




  

  

    
  


  

      

    @dnwe
dnwe



    mentioned this issue
    
      Aug 30, 2023
    





  


  

      
   Merged

  







  




  


  
    
  
  

    
  dnwe 

      added a commit
      that referenced
      this issue

    
      Aug 30, 2023
    
    

      


  

    

      


  

      
        @dnwe
  





      

        
          fix(client): force Event Hubs to use V1_0_0_0 (#2633)
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          ae5eee5
        
      

    

  

    

      
The problem with supporting Azure Event Hubs is that it isn't Apache
Kafka under the covers, it's an intermediate proxy that supports a
subset of the Kafka APIs at various versions and then maps them onto
Event Hubs protocol(s) at the backend. As as result Sarama's current
mechanism of specifying the KAFKA_VERSION to determine what protocol
versions to support and use doesn't really work properly with Event Hubs
because it supports an unusual set of protocols and even defines minimum
versions for ProduceRequest (v3) and FetchRequest (v4). For some reason
EventHubs is very behind on FetchRequest and MetadataRequest so the max
configuration you can use in Sarama is V1_0_0_0 and the minimum is
V0_11_0_0.

As we have recently bumped the default Version to V2_1_0_0 and support a
wider range of protocol versions, we're more likely to see issues raised
by Event Hubs users unless they have already pinned their version
correctly. To try and prevent this, attempt to detect use with Event
Hubs by inspecting the bootstrap broker addresses and overriding the
Version.

Contributes-to: #2470

Signed-off-by: Dominic Evans <dominic.evans@uk.ibm.com>

    




    

  









      

  
      



  

  @dnwe





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

      

      
            dnwe
  

      

      

      commented


        Aug 30, 2023


      
    


  





      


        


  
    

      
          
As discussed, pinning version to V1_0_0_0 manually would have prevented this issue when using Azure Event Hubs. However, it should also be resolved in v1.41.1 onward where we will force a pin to V1_0_0_0 if we think we're connecting to Event Hubs

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            

    

      
    

    


      

          @dnwe
dnwe




            closed this as completed


      Aug 30, 2023

    

  


    











  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

    None yet







      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  




              
No branches or pull requests







    
  




        
      

    

  
      

  

    

      2 participants
    

    

        
          @dnwe 
        
          @DeepanshuA