Unauthorized response in GetPackageRepositorySummaries #5999
Comments
|
For more context: we have an auth proxy that sits in front of kubeapps and our k8s apiserver. Here is our cluster configuration: kubeapps:
clusters:
- name: default
apiServiceURL: https://cloud-app-kubernetes-ingress.cloud/k8s
insecure: false
certificateAuthorityData: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvekNDQWVlZ0F3SUJBZ0lSQVA0VkppbWxoallaUnhkRjkyTC9MbVV3RFFZSktvWklodmNOQVFFTEJRQXcKR1RFWE1CVUdBMVVFQXhNT1kyOXlaWGRsWVhabExuUmxjM1F3SGhjTk1qTXdNakF5TVRVek1ERTRXaGNOTXpNdwpNakF5TURNek1ERTRXakFaTVJjd0ZRWURWUVFERXc1amIzSmxkMlZoZG1VdWRHVnpkRENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTGl5U3lYN0V1dWI0ZytiTjFxUzUrT1RnclNhT0xaeEFwdGMKNWhkUXpYRWgvOFQ2YkNMMTdEb2lKVkExeGlZdXJFeDdHRnpFVWZFWWgvbDAyRjV4V01LMDRjUU9OQ1htRlNiNQpJMFhXTXpTWWpOaFpVdHZCOW5VMWlwZkUwSnI0Q2g4MEhwOUlmN2RLQXd1dHlMR2oxWGk0b2lHVmV0OFJ0akQyCjdBdERWS1JRa1BOSzVTSElyZjNxQzFwMEZTa1VwZUJ5bXJFSGNCZEF0ZTZnOXlsZDh3cU8zR2RGZjZWSW5BSHcKL0szcUtRK1VVUVY0dXRqWklKZ0JzTit4Sy9zanN0OUVlYloxWjZOTGx5YXIyT2p3bE1DdFVrUkhFZXFkRFp0bQo3cmw4ajZ0dmQzL1NQY0NRT2NQeWwwSDVxa0J4L3hUK0I3b0dpZmVrbm96QUxzay85V0VDQXdFQUFhTkNNRUF3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkIveE5LTHkKaVVOSThIaGtJN0doZ2c4alI5QzJNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFoMFFOQ0ZuSVhVZ04vT2RXUApseGFScWk1cjU4VitxK245eTZZRHV3QmZYTUgreFFOdHphR3pTdm9TQ0paMTd2YVNheWxsTkdEV0hRUTJMU0tZClQ2ME1ZTE1aZnJHQVc5ZjVBampmek5ySURCdU4xKzZ1NzZ0T3BDcVBCeEEzMTB4LzhxRFEyeWJqL0p3N0w0WGwKSjR1OXlkeldXYjJZM3ZEOTVqTi9qNCt6dytIcUx5aGs2Mjg2MEJXOCtNakErMkNYVFJYdERUUGRRcTR2aGtrdQoyWlZ2L2pWUFFBeXQ0NW1XYkZQMVdqNlhEZlpvdzE5STRqTjM4ZkQ5L1I4TVFKMk1pZjNRb05oT2MzRlU1TnhnCmp6R0Q5dDN3US9kVHVCZy9jWHRPUEh1RlJZcGM3bXVlT1JkajMweWdrSTJwM0wwcm95LzZvekVzUkpzWU4vWW8KZnFlagotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==And here are logs from |
|
I noticed that the request is using context of I hard-coded those values to // GetPkgRepositories returns the list of package repositories for the given cluster and namespace
func (s *Server) GetPkgRepositories(ctx context.Context, cluster, namespace string) ([]*apprepov1alpha1.AppRepository, error) {
//resource, err := s.getPkgRepositoryResource(ctx, cluster, namespace)
resource, err := s.getPkgRepositoryResource(ctx, "default", "kubeapps")
if err != nil {
log.Infof("+[ERROR] helm GetPkgRepositories [%v]", err)
return nil, err
}
log.InfoS("+[MATT] s.getPkgRepositoryResource", "resource", resource, "ctx", ctx, "cluster", "default", "namespace", "kubeapps")
unstructured, err := resource.List(ctx, metav1.ListOptions{})
log.InfoS("+[MATT] GetPkgRepositories unstructured", "unstructured", unstructured, "err", err)Do you know why the frontend would be sending a gRPC request with that context Upon hardcoding I can view the catalog as expected, but cannot actually deploy a chart:
|
|
@absoludity could you take a look at this issue to properly triage it? Thanks! |
It's just terminology: we were adding support for other types of repositories (Carvel, Flux) and needed common terminology in the code, so followed the precedent that was used in other VMware packaging projects. I haven't been able to reproduce this yet, but do have one question: in your clusters config, why are you setting the I'll keep trying to repro... |
Gotcha, thanks for clarifying that.
We do this kind of hybrid approach (where we are as you point out really using the default cluster, but point to a different hostname via
Thanks @absoludity appreciate the efforts very much 🙏 Let me know if I can share anything else! |
Right, thanks for the explanation. So yes, I expect this is what's causing the difference in behavior. If you look at: kubeapps/dashboard/src/components/Catalog/Catalog.tsx Lines 254 to 264 in 72f49c5 the code is deciding there: if the current cluster is not the kubeapps/dashboard/src/actions/repos.ts Lines 88 to 91 in 72f49c5 So why is Kubeapps unable to determine the cluster on which Kubeapps is installed here when there is only one cluster defined in your config? Because it's actually possible to install Kubeapps on one cluster (say, a management cluster) while configuring the clusters so that users can only install to a different target cluster... and that is what Kubeapps has determined here. Which means the fix should be trivial: you should be able to simply add the kubeapps/chart/kubeapps/values.yaml Lines 1398 to 1400 in 72f49c5 Try setting that to true for your cluster and reload in your browser window and you should find that the request is sent with the correct context. Let me know if not. |
|
I just tried it quickly and it seems like due to setting
Specifically, now |
|
OK, which brings me to another question: what rules is your auth-proxy applying? I'm not sure, but suspect the gRPC-web request from the dashboard is not arriving at the kubeapps-apis service with the creds? (I was wondering about this in the earlier setup). I'll try to take a look tomorrow to see why any gRPC-web auth header could be stripped out, but it's not clear without knowing your auth-proxy. Or another possible (and better) way to find out: try setting up without your auth-proxy while still having an APIServiceURL set (maybe using the internal address). This will enable us to isolate whether it is a bug or an issue with the auth-proxy. |
|
So it looks like The behavior at our gateway (HAProxy) looks correct: it's getting the For kubeappsapis -> k8s request to succeed, the kubeappsapis backend would need to include those proxied request headers, like is done w/ the namespace filtering feature if you recall that one. I think this is the issue but am not 100%... |
It reads the |
|
Yep, that would not only make sense, but be very probably - as we don't have any e2e test for your special auth. So prior to our new API, the dashboard was making those requests directly to the k8s API (and in your case, via your proxy) so k8s would see it. OK, let me put a PR together and you can try the image. Thanks for the details - much easier now :) |
ppbaena
added
component/auth
|
Something like #6012 . Haven't tested it in real life yet though. |
…6016) <!-- Before you open the request please review the following guidelines and tips to help it be more easily integrated: - Describe the scope of your change - i.e. what the change does. - Describe any known limitations with your change. - Please run any tests or examples that can exercise your modified code. Thank you for contributing! --> ### Description of the change <!-- Describe the scope of your change - i.e. what the change does. --> After some testing of the previous PR, we found the issue is actually that kubeapps assumes the cluster on which Kubeapps is installed will never have an APIServiceURL set in the configuration (since it can be accessed via the in-cluster configuration at https://kubernetes.default). As it turns out, some users need to set the APIServiceURL of the cluster on which Kubeapps is installed because they use a proxy in front of the API server for authentication purposes, so it's important that Kubeapps also use this. ### Benefits <!-- What benefits will be realized by the code change? --> ### Possible drawbacks <!-- Describe any known limitations with your change --> ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - fixes #5999 ### Additional information <!-- If there's anything else that's important and relevant to your pull request, mention that information here.--> Signed-off-by: Michael Nelson <minelson@vmware.com>
We're trying to upgrade to kubeapps 2.6.3 (latest release) and I've encountered a strange bug when trying to load the
Catalog.Seemingly due to an Unauthorized GRPC response, the dashboard redirects back to
/and I end up right where I started (i.e. main dashboard page viewing applications).CatalogIs there some new configuration now that we need to do to permit our users to view package repositories?
We have 2
AppRepositoryconfigured in this cluster (BTW what is the difference to the "PackageRepository" concept?):The text was updated successfully, but these errors were encountered: