feat: template scheduler configs

[marwanad]

This change templates the proxy and candidate scheduler configs in values. It's useful for cases when a non-default plugin config is required.

[adrienjt]

In general, templated values are sort of an anti-pattern actually, and making all of the KubeSchedulerConfiguration a value removes a lot of the abstraction provided by the chart. In practice, what are you trying to change specifically? Could the plugins section be the value?

[marwanad]

@adrienjt the one thing I'm trying to change is appending a pluginConfig to adjust the scoringStrategy for NodeResourcesFit. The goal is to achieve as much binpacking as possible for the GPU resource.

I think we can probably do better, just by templating the pluginConfig section. I am planning on moving the hardcoded context durations to be part of the plugin arguments in another PR so this will play nicely with that change as well.

[adrienjt]

How would you test this? Can you for example set scoringStrategy in

admiralty/test/e2e/admiralty.sh

Lines 35 to 44 in b704bb7

	h $i upgrade --install admiralty charts/multicluster-scheduler -n admiralty --create-namespace -f $VALUES \
	--set controllerManager.image.repository=admiralty-agent \
	--set scheduler.image.repository=admiralty-scheduler \
	--set postDeleteJob.image.repository=admiralty-remove-finalizers \
	--set restarter.image.repository=admiralty-restarter \
	--set controllerManager.image.tag=$VERSION-amd64 \
	--set scheduler.image.tag=$VERSION-amd64 \
	--set postDeleteJob.image.tag=$VERSION-amd64 \
	--set restarter.image.tag=$VERSION-amd64
	k $i delete pod --all -n admiralty

and confirm that it works with an e2e test?