Revisit OTEL instrumentation with focus on links

[radeksimko]

Context

As part of #1327 we have instrumented various parts of the codebase to collect traces.

There we instrumented each job as a nested span, rather than root. While this reflects the relationships relatively accurately and this is helpful in the waterfall view in other contexts, it is not helpful for identifying jobs which take the most time as filtering capabilities are generally limited for nested spans.

Also, tracking time spent in the queue doesn't seem to be helping in this context either, as it clutters the waterfall view.

Proposal

• Replace the current span hierarchy rpc:* -> walk-path -> job spans with root spans:
  • rpc:* remains as-is
  • walk-path span becomes root span, with link back to rpc:* span
  • job-eval:* span becomes root span, with link back to walk-path and rpc:*
• Remove job and job-wait:* spans entirely

The assumption is that in any OTEL data visualising tool (such as Honeycomb), the root spans are queryable, unlike the nested ones, so we can more easily get a list of the slowest jobs and drill into those and still retain the context (what triggered these jobs).