POSTGRES_USER env var can't be set using kubernetes secrets

[dwarf-king-hreidmar]

Setup
Platform: K8s Arm64
Postgres Image: 16.1, 15.5, (presumably others)

What happens:
Using the contents of a k8s secret for POSTGRES_USER the variable is populated but no roles are created. Hopping into the pod with an exec shows the env vars populated correctly. However trying to connect with psql -U $POSTGRES_USER or psql -U postgres fails consistently no matter how many times I destroy and re-create the pod. I also deleted the pvc on every attempt to make sure I didn't have any data in the postgres data directory.

If I populate the POSTGRES_USER env variable with a hard coded value the install always succeeds and I can run psql -U $POSTGRES_USER

Here two bits of my k8s manifest to compare:

Not Working:

        env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: nextcloud-postgres
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nextcloud-postgres
                  key: password

Working

        env:
            - name: POSTGRES_USER
              value: nextcloud
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nextcloud-postgres
                  key: password

What I expect to happen:
It doesn't matter how I set the environment variables and I can use a secrete if I want to.

I can hardly believe this and I'm happy to test anything you folks want to get to the bottom of it. It makes me wonder if the POSTGRES_USER needs to be populated really early in the deployment and there is some sort of race condition

Lastly. I've used this method for building up k8s configs before, even on the postgres image. I have some running already in the same cluster.

[tianon]

I'm guessing you somehow already have an initialized data volume (this image will only initialize the volume if it isn't already initialized). Can you try making sure you absolutely have a definitely fresh data volume / folder and try again?

[dwarf-king-hreidmar]

i sure can. I run kubectl delete pvc -n $ns and $pvc and clears it. I suppose there is some possibility that some how on the backend the ceph volumes are getting reused but that would be wacky. I'll just change the name of the volume to be sure

[dwarf-king-hreidmar]

So this is what I found:

1. I started with a kinda working postgres. It had a user "nextcloud" and I could run psql -U nextcloud and I was in. This happened because I changed the POSTGRES_USER var as I mentioned above.
2. I deleted the postgres resources (service and stateful set)
3. I deleted the pvc
4. I created a new PVC with a slightly different name
5. I edited my postgrs volume to mount the new pvc
6. I started the postgres resources and again I get role "nextcloud" doesn't exist.
7. Delete postgres resources
8. Change back to directly setting POSTGRES_USER
9. Delete the pvc again
10. start postgers resources
11. I can log in as nextcloud locally with psql -U nextcloud

[octavian2204]

+1, same here. (testing on kind version 0.24.0)

works:

          env:
            - name: POSTGRES_USER
              value: psql-user

doesn't:

          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: env-secrets
                  key: POSTGRES_USER

secret:

kind: Secret
metadata:
  name: env-secrets
type: Opaque
data:
  POSTGRES_USER: ZGV2LXVzZXIK

UPDATE:
fixed my issue. The problem is that my base64 hash contained a new line at the end:

# with new line:
~ echo 'ZGV2LXVzZXIK' | base64 -d
dev-user

# w/o new line:
~ echo 'ZGV2LXVzZXI=' | base64 -d
dev-user%

the problem is that working in console echo sends the string WITH the new line at the end:

~ echo 'dev-user' | base64
ZGV2LXVzZXIK

to send w/o the new line, the command should be:

~ echo -n 'dev-user' | base64
ZGV2LXVzZXI=