Ingester search production readiness

[mdisibio]

Ingester search was recently merged #806 but it is marked experimental. This issue is tracking the list of additional work to consider it production-ready:

• Config: Ingester: Make search block encoding and page size configurable (Expose config options for search block encoding and page size #990)
• Config: Querier: separate search timeout from trace lookup timeout (Split up QueryTimeout for trace and search queries #984)
• Querier: tolerate disappearing ingesters and return partial results
  • Related to Support Returning Partial Results #899
• Querier/Ingester: correctly enforce limit, make the default configurable (Make default search limit configurable #1022)
• Ingester: Add block headers to eliminate blocks from search faster. Maybe also use blockheaders for autocomplete lookups instead of in-memory LRU cache which is somewhat inefficient. (Add search block headers #943)
• Ingester: Fix duration filtering (sometimes incorrect results are returned) (Fix 32/64-bit issue in search duration filtering #933)
• Ingester: Dedupe WAL records when converting StreamingSearchBlock -> to BackendSearchBlock (Dedupe search records while replaying WAL #940)
• Ingester: Reload search blocks on startup (requires gracefully handling blocks without search data) (Reload search blocks and replay search WAL #1000)
• Distributor: Extract span status and make available for searching (error=y?) (Support error and status.code searches #1093)
• Operational: alerts for search apis
• Operational: dashboard panels for search apis (Add latency and qps panels for search to operational dashboard #930)
• Documentation: config options, apis, alerts, troubleshooting (Document search apis and remaining config options #1003)

[joe-elliott]

We are seeing traceheaders use quite a bit of disk space in the ingesters. More than previously estimated. Consider the following tasks as well:

• Global denylist / per-tenant allowlist of tags to be dropped / added to traceheaders (Support global denylist and per-tenant allowlist of tags for search data #960)
• Add a max tag value length. Just truncate the value if it's > then some configurable number of bytes/runes

[joe-elliott]

Two issues we've seen in ops at ~1.5M spans/second and searching continuously:

• Twice in the past 7 days this has been logged:
  a117bf98-f9d7-4429-8023-54fe82ca18f5 read /var/tempo/wal/search/a117bf98-f9d7-4429-8023-54fe82ca18f5:1:searchdata: file already closed
  This is due to a race condition in searchWal. Determine how to bubble up this error correctly or avoid.
• Occassionally ingester goroutines spike in the 10s of thousands and search gets very slow. This resolves after a few seconds, but we should research this and attempt to prevent it.
  

[mdisibio]

Two issues we've seen in ops at ~1.5M spans/second and searching continuously:

#1033 definitely addresses the first issue, and likely the 2nd. When a deadlock occurs and the ingest path is stalled, many goroutines are created for queued up ingest traffic which continues to be received.

[mdisibio]

• Occassionally ingester goroutines spike in the 10s of thousands and search gets very slow. This resolves after a few seconds, but we should research this and attempt to prevent it.

This was improved already with #1033 but #1076 further reduces contention between search and ingest traffic.