csi plugins receiving invalid/incorrect authority header

[travisghansen]

Environmental Info:
K3s Version: present in all known versions

Node(s) CPU architecture, OS, and Version: NA

Cluster Configuration: NA

Describe the bug:

grpc-go in certain circumstances sends the path to the uds as the host/authority header. This is seemingly non-compliant and many http2 servers outright reject the request.

I've created a project here to circumvent the issue: https://github.com/democratic-csi/csi-grpc-proxy

Further discussion here: democratic-csi/democratic-csi#164

Steps To Reproduce:

• Installed K3s:

helm repo add democratic-csi https://democratic-csi.github.io/charts/
helm repo update

helm upgrade --install \
--create-namespace \
--namespace democratic-csi \
--version 0.11.1 \
-f local-hostpath.yaml \
local-hostpath democratic-csi/democratic-csi

## local-hostpath.yaml
# driver only works with 1.16+
csiDriver:
  # should be globally unique for a given cluster
  name: "org.democratic-csi.local-hostpath"
  storageCapacity: true
  fsGroupPolicy: File

storageClasses:
- name: local-hostpath
  defaultClass: false
  reclaimPolicy: Delete
  volumeBindingMode: WaitForFirstConsumer
  # distributed support is not yet ready
  allowVolumeExpansion: false
  parameters:

  mountOptions: []
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:

# if your cluster supports snapshots you may enable below
volumeSnapshotClasses: []
#- name: nfs-client
#  secrets:
#    snapshotter-secret:


controller:
  enabled: true
  strategy: node
  
  externalProvisioner:
    extraArgs:
    - --leader-election=false
    - --node-deployment=true
    - --node-deployment-immediate-binding=false
    - --feature-gates=Topology=true
    - --strict-topology=true
    - --enable-capacity=true
    - --capacity-ownerref-level=1

  # distributed support is not yet ready
  externalResizer:
    enabled: false

  externalSnapshotter:
    enabled: true
    extraArgs:
    - --leader-election=false
    - --node-deployment=true

node:
  driver:
    #image: democraticcsi/democratic-csi:latest
    #image: democraticcsi/democratic-csi:next
    image: democraticcsi/democratic-csi:v1.6.1
    imagePullPolicy: Always
    logLevel: debug

    extraVolumeMounts:
    - name: local-storage
      mountPath: /var/lib/csi-local-hostpath
      mountPropagation: Bidirectional

  extraVolumes:
  - name: local-storage
    hostPath:
      path: /var/lib/csi-local-hostpath
      type: DirectoryOrCreate


driver:
  config:
    #driver: 
    # rest of per-driver config data/syntax
    driver: local-hostpath
    instance_id:
    local-hostpath:
      # generally shareBasePath and controllerBasePath should be the same for this
      # driver, this path should be mounted into the csi-driver container
      shareBasePath:      "/var/lib/csi-local-hostpath"
      controllerBasePath: "/var/lib/csi-local-hostpath"
      dirPermissionsMode: "0777"
      dirPermissionsUser: root
      dirPermissionsGroup: root


# change to true to deploy the workaround
csiProxy:
  enabled: false

Expected behavior:

Driver comes up healthy and is properly registered with kubelet etc.

Actual behavior:

kubelet fails to register the driver. Error looks something like this:

I0321 14:55:30.133080       1 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR,}
E0321 14:55:30.133106       1 main.go:122] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR, restarting registration container.

Additional context / logs:

Backporting

• Needs backporting to older releases

[brandond]

I'm confused, is this a K3s specific issue, or did you report it here because that's where you reproduced it? This seems like something that should be tracked in the grpc-go, kubernetes, or democratic-csi projects.

Is there anything you believe we could do to resolve this problem within K3s?

[travisghansen]

@brandond the problem was supposedly resolved in grpc-go version ~1.31. I'm unclear why the behavior is still exhibited even when the client is newer. The fix will indeed be something like this: hashicorp/nomad#12359

[brandond]

Yeah, HTTP over Unix sockets is weird for sure. With the unix:// URI scheme there isn't a host (authority) portion of the URL, so it's unclear what it should be set to, and it seems like not all libraries agree on that or handle it properly.

[travisghansen]

From what I've seen the common consensus seems to be setting the value to localhost for uds connections. I don't know if that is technically right/wrong but seems to be what several of the clients do.