Increase resource defaults for monitoring stack

[blancharda]

Is your feature request related to a problem? Please describe.

The monitoring stack (prometheus, grafana, loki etc) have enough resources to start, but often struggle when scaled beyond a single node or with higher volume workloads. We should consider updating the default values, and provide clear guidance on suggested overrides for various deployment scales/sizes.

Additional context

Prometheus in particular seems to struggle even with relatively small workloads.

[mjnagel]

This may be a good reason to evaluate scalable/HA grafana + prometheus. For reference on DUBBD in the past we had tickets for HPAs on those two and noted necessary external dependencies:

• Configure HPA for Prometheus uds-package-dubbd#369
• Configure HPA for Grafana uds-package-dubbd#368

Loki itself defaults to a scalable mode (but single replica) with no resource limits/requests.

I think we should definitely:

1. Document the overrides for scaling these up (resources as a first pass, replicas/hpa as we further explore/support those).
2. Identify any upstream guidance on sizing/scaling for each.
3. As we gather more data from CI/staging environments also document our own suggested sizing based on unique core needs.