Make scraping Kubernetes API from default Agent installation optional

[rfratto]

The default Kubernetes manifest for the Agent makes too many assumptions about what the users might want to scrape. In particular, it immediately scrapes the Kubernetes API, which is the source of a significant amount of active series (at least 30K per API server).

This should be made optional, defaulting to not scraping the Kubernetes API.

For those looking into contributing this themselves, the change must happen within the Tanka environment used to generate the manifest.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[MrSaints]

+1 for this.

We decided to give Grafana Cloud a try, but the default instructions for installing the Agent on Kubernetes captures 20k+ metrics. This instantly hits a 429 (rate limit) status code, and exceeds what the Pro plan offers. From running the agent for about less than a minute, we already have a projected monthly spend of $300+ which doesn't seem right at all. And this is on a sandbox cluster with very little workload.

[rfratto]

we already have a projected monthly spend of $300+ which doesn't seem right at all. And this is on a sandbox cluster with very little workload.

Yeah, sorry about this 😞 I would personally agree that a small cluster shouldn't cost $300+ to monitor. We're working on our support for Kubernetes over the next few months and improving this will be a major part of that.

If you're looking for a quick hacky workaround, you can scale deployments/grafana-agent-deployment to 0 replicas, which will only disable the Kubernetes API job. Just be aware if you run the default installation script again, it'll be scaled back up to 1 replica.

[rfratto]

Closed by #384