LLM Semantic Conventions: Initial PR

[nirga]

Work towards #327.

Changes

Replacing PRs #639 and #483. Introduces semantic conventions for modern AI systems, following discussions in OpenTelemetry LLM semconv working group.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• schema-next.yaml updated with changes to existing conventions.

[iag]

Thanks for this initial revision!

[chalin]

LGTM for the issues that I had raised earlier.

[joaopgrassi]

@nirga are you able to fix the conflicts? I think this looks good to merge :)

[nirga]

Thanks @joaopgrassi! I've resolved all the comments and fixed the conflicts. I think we're good to go :)

[alolita]

Changes lgtm.

A call out I have is that while we specifically only mention LLMs - ML models have been around for a long time - so isn't model observability including LLMs what we want to address with these semantic conventions? Lets discuss in our weekly SIG meeting.

[cartermp]

FWIW my take is that scoping to LLMs for now is still the right call, but we should absolutely not design for only them (and I believe that's not happeing). Even in this space, there's multi-modality, language models (like embedding models) that aren't about generating text, diffusion models, etc.

The good news is that a lot of the fundamentals around capturing data for various purposes are likely to be similar across many different models types.