Refactor semantic conventions

[tedsuo]

Based on discussions within the Semantic Conventions SIG, I am proposing a refactor to how the semantic conventions are laid out within the spec.

Proposed changes:

• Centralize the location of all semantic conventions, rather than split them across many subsections of the spec.
• Create one folder per topic (HTTP, faas, messaging, etc). All relevant tracing, metric, and resource conventions are combined into these topics.
• Move all implementation-specific conventions (HBase, AWS lambda, etc) to an /implementation folder under each topic.

The new layout can viewed here, on the branch in GitHub.

Example layout:

-- semantic_conventions/
   -- database/
        database.md
        sql.md
        --implementations/
            cassandra.md
            dynamodb.md
            hbase.md
            mongodb.md
            mysql.md
            redis.md

Goals:
This new layout should make it easier to browse and comprehend all of the available semantic conventions. This should also make it easier to assign domain experts as codeowners for each convention. It also makes it easier to see how most implementations are underspecified. This will help the Semantic Conventions / Instrumentation SIG move forwards.

This PR only attempts to take separate and recombine the concepts currently described in the spec, while changing the wording as little as possible. It does not attempt to adjust any concepts or clarify existing any descriptions. Once the refactor is complete, approvers should be assigned to each convention so that we can begin processing improvements to the spec.

This PR is currently a draft, but it is ready for initial feedback.

Completed so far:

• Refactored trace semantics

TODO:

• Refactor metric semantics
• Refactor resource semantics

[pyohannes]

Looks like a great start!

[pyohannes]

You probably want to name the directory rpc and not rpc.md.

[pyohannes]

I like having implementations in separate documents. This will allow us to have experimental documents for implementations, while the overall conventions could be stable.

[lmolkova]

Looks great!

What about yaml files? It looks like many changes should be done through them and the instrumentation SIG expertise area should cover both yaml and md files

[lmolkova]

I suggest keeping yaml and md together.
Since we probably want to cover metrics later on, would it make sense to keep current conventions in the trace folder?

-- semantic_conventions/
   -- database/
       -- trace/
           database.yml
           database.md
           sql.md
           --implementations/
                  cassandra.md
                  ...

[tedsuo]

@lmolkova I'm not familiar with how the yaml files work, I thought they were generated from the markdown.

I agree we need to make them work properly before committing this. But if we want to change their behavior/location, perhaps that can be a follow up PR? I don't know who currently consumes them.

[lmolkova]

@lmolkova I'm not familiar with how the yaml files work, I thought they were generated from the markdown.

I agree we need to make them work properly before committing this. But if we want to change their behavior/location, perhaps that can be a follow up PR? I don't know who currently consumes them.

md files (<semconv> blocks) are generated from yaml files with this tool https://github.com/open-telemetry/build-tools/tree/main/semantic-conventions#usage.

Anyway, I'm fine with a follow-up PR.

[Oberon00]

I have heard the misconception that the YAML is generated from the markdown for the second time from different persons now. But I wonder how this occurs. Isn't it be far more usual to generate text from structured data than the other way round? Is there something we could change to make it more obvious how this works, maybe in the syntax of the marker comments, or elsewhere?

There is documentation about these also in the spec BTW: https://github.com/open-telemetry/opentelemetry-specification/tree/main/semantic_conventions

[tedsuo]

@lmolkova for metrics vs trace, I want to see if we can combine them rather than keep them separate. As an instrumentation author, you will need to understand the whole model - configuration, span structure, and how metrics relate to these.

Some semantics, like faas, have also have resource definitions. I want to see if we can present this as a single model.

My hope is that by moving out all of the implementation-specific details, there will now be room to keep all of this info in a single file. If not, I'd like to try using suffixes like -trace.md and -metric.md to keep the all of the details next to each other in the same folder.

[jamesmoessis]

Amazing! I really like the idea of bringing all semantic convention related things together, and of course splitting the implementation specific conventions out. This will help the semantic convention towards stability.

The YAML is probably the first source of truth that needs changing to make this structure a reality, as others have mentioned. I also imagine that a fair amount of re-work would be required in build-tools to generate this structure. I am very happy to help out with these pieces of work.

[kenfinnigan]

I like the layout and its separation of implementations, and how all signal conventions are in one place.

However, as @lmolkova mentioned, this should also be done for the YAML files otherwise it will be very confusing to find the right file to change

[jsuereth]

JDMC or JDBC?

Also, so I understand, "database" implies database-clients and database-servers?

[jamesmoessis]

I thought it was just clients, and the span kind is CLIENT.

[jsuereth]

Was the directory mean to be rpc or rpc.md?

[jsuereth]

Like this shift!

[carlosalberto]

Telemetry signals? Otherwise you will have to list Logs (and other stuff) in the future.

[carlosalberto]

Great work, thanks for working on this!

[joaopgrassi]

Is there something we could change to make it more obvious how this works, maybe in the syntax of the marker comments, or elsewhere?

@Oberon00 I think if the suggestion structure in this PR gets accepted/merged, the top-level README there should probably point out that the .md files here are "auto-generated" by the yaml files. I think maybe that will help already? Another thing would be to put also the yaml files together in the same folder, but I don't know if that's even possible.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[jamesmoessis]

Hey folks, I've raised a PR to refactor the YAML so it follows the same structure as this. This is a pre-requisite of refactoring the markdown like this PR shows.

#2019

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[tigrannajaryan]

@tedsuo any chance we can revive this? I think this is very important.