[DOC] Update Helm chart doc

[knylander-grafana]

Update the Helm chart documentation to address gaps in content and common questions.

Helm chart doc: #4326

pages to update:

• Helm chart doc, https://grafana.com/docs/helm-charts/tempo-distributed/next/get-started-helm-charts/
• Enterprise traces gateway, https://grafana.com/docs/enterprise-traces/latest/configure/gateway/

Related:

• Requires engineering work - https://github.com/grafana/enterprise-traces/issues/1122
• Requires engineering work - Update Helm chart for Alloy - https://github.com/grafana/tempo-squad/issues/440
• tempo-distributed helm install fails if gateway:enabled:true helm-charts#3427
• https://github.com/grafana/enterprise-traces/issues/814
• [DOC] Update overrides content in Helm chart to clarify configuration for new, legacy #3820

Tasks

Preview Give feedback

1. Evaluate helm chart doc to make sure steps are up to date
2. Evaluate if we have gaps
3. Add TLS info - Add doc about configuring TLS with Helm #4328
4. Clarify the different types of gateway used in Helm chart for GET (enterpriseGateway), Tempo (gateway: true), nginx
5. Use the collapse shortcode for large configuration sections (https://grafana.com/docs/writers-toolkit/write/shortcodes/#collapse)
6. Address issues found by FE walkthrough of Helm chart docs (see comments below)
7. Check for streaming endpoint in Helm chart [tempo-distributed, grafana] How to enable traceQLStreaming getting error "rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"" #3987
8. [DOC] Update overrides content in Helm chart to clarify configuration for new, legacy #3820
   keepalive type/docs +2

[knylander-grafana]

Zach's PR to handle TLS with Helm - #4328

[hedss]

Just rough C&P from GET install today:

• 4 cores and 16GB memory (don't believe this, unless it's 4vCPU per core, and anti-affinitising components could actually fail here if there aren't enough nodes)
• Image tag shouldn't be set because for latest chart version options will be wrong, latest is always defined as App Version anyway)
• Don't need to enable enterpriseGateway if enterprise is enabled, not even sure this flag is still valid?
• Don't need to explicitly disable gateway (it's never on by default) anyway
• Minio isn't enabled by default so you don't need to disable it in the chart if not using it
• Add an example of both Tempo and GET gateways for ingress (it's not clear how these are different and has caused confusion). We should be very clear on what each does and how to configure it. I don't think we should be explicitly stating the Enterprise Gateway by default as this could add an ingress when only an internal cluster/shared VPC setup is required.

Worth checking the rest of the doc too for other things when we go through this.
cc @ayah-el @danstadler-pdx

[knylander-grafana]

Verify that the GRPC endpoint for streaming is listed. Http2 proxy.
Related: #3987 (comment)

From Joe:
I have not operated istio so I can't say for sure. You will need to make sure that envoy is proxying HTTP2 generically or perhaps it supports GRPC more specifically?

Yes it seems there is missing config in the nginx gw in the helm chart to allow for TraceQL streaming. I'm was looking at the config here and I don't see a pass for the streaming GRPC endpoint. It may also need special config for proxying HTTP2.

[knylander-grafana]

PR to address FE issue:#4370

[knylander-grafana]

Overrides doc updated in this PR: #4415