feat: allow ssh-credentials to be set in agent and plugin

[42atomys]

Description

This pull request introduces the capability to configure ssh credentials directly in the Buildkite agent and plugin settings. This feature enhances security and flexibility by allowing users to set up git access credentials, improving the management and use of private repositories in CI/CD workflows.

This pull request put gitFromEnv as deprecated because this is not relevante to the usage (currently this field is used to set a ssh credentials and can be also used as envFrom override). The gitFromEnv still working for backward compatibility but as the project are not stable yet, what is the vision about deprecated lifecycle ?

⚠️ DEPRECATED ⚠️

The method for setting git credentials in Buildkite agents stack and k8s plugins is changing.
Please update your configurations to use the new method introduced in PR #248 for enhanced security and flexibility.

# DEPRECATED WAY
gitEnvFrom:
- secretRef: { name: agent-stack-k8s }
# REPLACEMENT
ssh-credentials-secret: agent-stack-k8s

[42atomys]

Hi @triarius, We start the pod spec at the agent level with the ssh-credentials, other pull request will come today with the pod-spec-patch feature 🚀

I have multiple question about the future of this agent-k8s-stack and the vision to be align with it, can we discuss via Slack ?

[triarius]

Hi @42atomys, sure, but our timezones might make synchronous conversation hard. It might be best to email support@buildkite.com, and they can arrange some channel for us to communicate.

[triarius]

I've run the CI pipeline on your branch, @42atomys. There are some failures: https://buildkite.com/buildkite-kubernetes-stack/kubernetes-agent-stack/builds/1040#018d97ce-c014-46c9-8704-516f22867af4

[42atomys]

@triarius I will send a message today to support@buildkite.com and sorry about the errors, a stash still on my computer. My bad

Currently on the last failling test

[42atomys]

@triarius Trying to run integration test in an empty organization, but job is not taked by the agent, I dont found the source. I found an issue and resolve it. Sorry for the test approval train 🙏

[peterkracik]

this would be really helpful 🙏

[triarius]

@42atomys I found some time to look at this again. I think its mostly a much-needed improvement. But I have a few quibbles:

1. The tests were failing because there was a mix-up between the kebab case needed for ssh-credentials-secret when it's in the config (and the helm values.yaml) and the camel case needed when it's in the Buildkite Pipeline YAML. I've suggested a fix, but it seems this situation is rife for confusion in the future.

2. I'm not sure if we want to specify checkout secrets at the step level at all. Every step in a pipeline will typically need the same git secret, as every step will typically checkout the same git repo.

Given these, I'm inclined to drop the ability to specify sshCredentialsSecret in the kubernetes plugin, and just rely on what's in the controller config.

I've made a branch where I've built on your changes and obtained a green build: main...triarius/allow-git-credentials-as-secret-from-agent-and-plugin

If you're happy for me to push this to your PR branch, I'm happy to merge this as is.

Otherwise, LMK if you need the per-step sshCredentialsSecret. I think there's already at least 2 ways to get this into a containers, so it's not needed to add a third, especially as I think it's not that useful to do it on a per-step basis.

[triarius]

Suggested change

	ssh-credentials-secret: agent-stack-k8s
	sshCredentialsSecret: agent-stack-k8s

[triarius]

Suggested change

	ssh-credentials-secret: agent-stack-k8s
	sshCredentialsSecret: agent-stack-k8s