Vault CSI provider not making use of received token TTL

[lukpep]

This problem is connected to secrets auto-rotation but not only - this only makes it visible.
When enabled this makes CSI provider ask Vault about secrets every 2 minutes (by default - can be changed with rotation-poll-interval parameter) to keep secrets in k8s in sync with the vault - pretty obvious.
Problem is that vault CSI provider is not respecting token TTL (which in standard config promoted by docs is 768h - 32 days) and religion every 2 minutes which can be seen on Nginx ingress logs:

ingress-nginx-controller-6c9594575f-2hll2 controller 192.168.130.1 - - [20/Mar/2022:11:51:33 +0000] "POST /v1/auth/k8s-apps-prod/login HTTP/2.0" 200 710 "-" "Go-http-client/2.0" 1363 0.494 [vault-in-cluster-vault-active-8200] [] 192.168.128.49:8200 710 0.494 200 b7efe65cb03bc79fc0218dac8a32d1be
ingress-nginx-controller-6c9594575f-2hll2 controller 192.168.130.1 - - [20/Mar/2022:11:51:33 +0000] "GET /v1/app-secrets/data/some-random-app/prod HTTP/2.0" 200 2938 "-" "Go-http-client/2.0" 71 0.002 [vault-in-cluster-vault-active-8200] [] 192.168.128.49:8200 2950 0.002 200 0a2383fd543179ef56ae37d335a70ceb
--
ingress-nginx-controller-6c9594575f-2hll2 controller 192.168.130.1 - - [20/Mar/2022:11:53:33 +0000] "POST /v1/auth/k8s-apps-prod/login HTTP/2.0" 200 710 "-" "Go-http-client/2.0" 1363 0.447 [vault-in-cluster-vault-active-8200] [] 192.168.128.49:8200 710 0.446 200 f1b278a2f151da20ff84bc4d1451ded0
ingress-nginx-controller-6c9594575f-2hll2 controller 192.168.130.1 - - [20/Mar/2022:11:53:33 +0000] "GET /v1/app-secrets/data/some-random-app/prod HTTP/2.0" 200 2938 "-" "Go-http-client/2.0" 72 0.002 [vault-in-cluster-vault-active-8200] [] 192.168.128.49:8200 2950 0.001 200 15d15e99cfc6afb015550dfcc0089d09
--
ingress-nginx-controller-6c9594575f-2hll2 controller 10.97.128.107 - - [20/Mar/2022:11:55:33 +0000] "POST /v1/auth/k8s-apps-prod/login HTTP/2.0" 200 710 "-" "Go-http-client/2.0" 1363 0.588 [vault-in-cluster-vault-active-8200] [] 192.168.128.49:8200 710 0.587 200 28e02dea3d629bf28c4ca522f5a82136
ingress-nginx-controller-6c9594575f-2hll2 controller 10.97.128.107 - - [20/Mar/2022:11:55:33 +0000] "GET /v1/app-secrets/data/some-random-app/prod HTTP/2.0" 200 2938 "-" "Go-http-client/2.0" 71 0.001 [vault-in-cluster-vault-active-8200] [] 192.168.128.49:8200 2950 0.002 200 0e734c6f5a1464a8d072fad83bfb1ddd

This results in a huge number of lease objects created in the vault - which are also kept in memory I believe.
auth method configuration:

Expected behaviour:
TTL should be respected and CSI driver should not login every rotation-poll-interval - received tokens should be probably cached and reused.

Tested on 0.4.0 and on 1.0.0 - same behaviour
This issue is a result of a discussion from here
I will create a separate issue for the second topic - per pod secret sync.

[bobmhong]

We are experiencing the same issue with a high number of new leases created due to the Vault CSI provider and would appreciate having this issue prioritized. Reducing the default TTL to 30 minutes has helped, but until this is addressed, we will not recommend the Vault CSI provider for further use.

[tomhjp]

Thanks for opening this with lots of details. It's still a work in progress, but I'd like to combine #163 and hashicorp/vault-helm#749 to address a lot of the pain points. i.e. the CSI Provider's login logic will stay largely the same, but Vault Agent will be a caching intermediary between the CSI provider and Vault.

[Rosslu]

I have the same issue, running 1.2.0

Seems every 2 minutes the password gets rotated and there is s huge number of leases.

[vistrcm]

same issue for us

[zeppelinen]

If anything, my current workaround is to use vault-secrets-operator.

[tomhjp]

Thanks for the reports and +1s all. #202 will cache Vault client tokens in-memory, which should fix this when combined with the Vault Agent sidecar that hashicorp/vault-helm#749 added for lease caching and renewals.