do not request unused cpu resources

[jfcoz]

do not request unused 100m cpu resources when VPA is enabled

[saumas]

@jfcoz, what's the purpose of this change? CPVPA is set up to scale only memory. There's no VPA, only CPVPA. 100m is already super low. In larger clusters the active agent replica uses about 20 millicpus when collecting deltas, but it needs a lot more when booting up and collecting the initial snapshot, so 1m might slow it down to a crawl and it will hang.

[jfcoz]

I see 1m usage on small nodes.
If everybody set 100m by default, for castai, all daemonset of all cloud providers tools by default, there is nothing left for apps.
I you don’t want to hardcode another value, at least add an option to allow to configure it.

[saumas]

@jfcoz, could you change the PR to allow configuration of requests and limits?