add remaining url fields to the registry

open-telemetry · Nov 9, 2023 · 0ea2294 · 0ea2294
1 parent 65a3cc7
commit 0ea2294
Show file tree

Hide file tree

Showing 3 changed files with 111 additions and 14 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,8 @@ release.
 ### Breaking
 
 ### Features
+- Add remaining ECS fields to the url namespace
+  ([#495](https://github.com/open-telemetry/semantic-conventions/pull/495/))
 
 ### Fixes
 

diff --git a/docs/attributes-registry/url.md b/docs/attributes-registry/url.md
@@ -8,15 +8,36 @@ linkTitle: URL
 <!-- semconv registry.url(omit_requirement_level) -->
 | Attribute  | Type | Description  | Examples  |
 |---|---|---|---|
+| `url.domain` | string | Domain of the url, such as "opentelemetry.io". [1] | `www.foo.bar`; `opentelemetry.io`; `3.12.167.2`; `[1080:0:0:0:8:800:200C:417A]` |
+| `url.extension` | string | The file extension from the original request url, excluding the leading dot. [2] | `png`; `gz` |
 | `url.fragment` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>The [URI fragment](https://www.rfc-editor.org/rfc/rfc3986#section-3.5) component | `SemConv` |
-| `url.full` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>Absolute URL describing a network resource according to [RFC3986](https://www.rfc-editor.org/rfc/rfc3986) [1] | `https://www.foo.bar/search?q=OpenTelemetry#SemConv`; `//localhost` |
+| `url.full` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>Absolute URL describing a network resource according to [RFC3986](https://www.rfc-editor.org/rfc/rfc3986) [3] | `https://www.foo.bar/search?q=OpenTelemetry#SemConv`; `//localhost` |
+| `url.original` | string | Unmodified original url as seen in the event source. [4] | `https://www.foo.bar/search?q=OpenTelemetry#SemConv`; `search?q=OpenTelemetry` |
 | `url.path` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>The [URI path](https://www.rfc-editor.org/rfc/rfc3986#section-3.3) component | `/search` |
-| `url.query` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>The [URI query](https://www.rfc-editor.org/rfc/rfc3986#section-3.4) component [2] | `q=OpenTelemetry` |
+| `url.port` | int | Port of the request | `443` |
+| `url.query` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>The [URI query](https://www.rfc-editor.org/rfc/rfc3986#section-3.4) component [5] | `q=OpenTelemetry` |
+| `url.registered_domain` | string | The highest registered url domain, stripped of the subdomain. [6] | `example.com`; `foo.co.uk` |
 | `url.scheme` | string | ![Stable](https://img.shields.io/badge/-stable-lightgreen)<br>The [URI scheme](https://www.rfc-editor.org/rfc/rfc3986#section-3.1) component identifying the used protocol. | `https`; `ftp`; `telnet` |
+| `url.subdomain` | string | The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain. [7] | `east`; `sub2.sub1` |
+| `url.top_level_domain` | string | The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is `com`. [8] | `com`; `co.uk` |
+| `url.username` | string | Username of the request. | `username` |
 
-**[1]:** For network calls, URL usually has `scheme://host[:port][path][?query][#fragment]` format, where the fragment is not transmitted over HTTP, but if it is known, it SHOULD be included nevertheless.
+**[1]:** In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the domain field.  If the URL contains a [literal IPv6 address](https://www.rfc-editor.org/rfc/rfc2732#section-2)  enclosed by `[` and `]`, the `[` and `]` characters should also be captured in the domain field.
+
+**[2]:** The file extension is only set if it exists, as not every url has a file extension. When the file name has multiple extensions `example.tar.gz`, only the last one should be captured `gz`, not `tar.gz`.
+
+**[3]:** For network calls, URL usually has `scheme://host[:port][path][?query][#fragment]` format, where the fragment is not transmitted over HTTP, but if it is known, it SHOULD be included nevertheless.
 `url.full` MUST NOT contain credentials passed via URL in form of `https://username:password@www.example.com/`. In such case username and password SHOULD be redacted and attribute's value SHOULD be `https://REDACTED:REDACTED@www.example.com/`.
 `url.full` SHOULD capture the absolute URL when it is available (or can be reconstructed) and SHOULD NOT be validated or modified except for sanitizing purposes.
 
-**[2]:** Sensitive content provided in query string SHOULD be scrubbed when instrumentations can identify it.
+**[4]:** In network monitoring, the observed URL may be a full URL, whereas in access logs, the URL is often just represented as a path. This field is meant to represent the URL as it was observed, complete or not.
+`url.original` MUST NOT contain password passed via URL in form of `https://username:password@www.example.com/`. In such case password SHOULD be redacted and username might be redacted and attribute's value SHOULD be `https://username:REDACTED@www.example.com/`.
+
+**[5]:** Sensitive content provided in query string SHOULD be scrubbed when instrumentations can identify it.
+
+**[6]:** This value can be determined precisely with the [public suffix list](http://publicsuffix.org)
+
+**[7]:** The subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+
+**[8]:** This value can be determined precisely with the [public suffix list](http://publicsuffix.org).
 <!-- endsemconv -->
diff --git a/model/registry/url.yaml b/model/registry/url.yaml
@@ -4,11 +4,30 @@ groups:
     type: attribute_group
     prefix: url
     attributes:
-      - id: scheme
+      - id: domain
+        type: string
+        brief: >
+          Domain of the url, such as "opentelemetry.io".
+        note: >
+          In some cases a URL may refer to an IP and/or port directly,
+          without a domain name. In this case, the IP address would go to the domain field. 
+          If the URL contains a [literal IPv6 address](https://www.rfc-editor.org/rfc/rfc2732#section-2) 
+          enclosed by `[` and `]`, the `[` and `]` characters should also be captured in the domain field.
+        examples: ["www.foo.bar", "opentelemetry.io", "3.12.167.2", "[1080:0:0:0:8:800:200C:417A]"]
+      - id: extension
+        type: string
+        brief: >
+          The file extension from the original request url, excluding the leading dot.
+        note: >  
+          The file extension is only set if it exists, as not every url has a file extension.
+          When the file name has multiple extensions `example.tar.gz`, only the last one should be captured `gz`, not `tar.gz`.
+        examples: [ "png", "gz" ]
+      - id: fragment
         stability: stable
         type: string
-        brief: 'The [URI scheme](https://www.rfc-editor.org/rfc/rfc3986#section-3.1) component identifying the used protocol.'
-        examples: ["https", "ftp", "telnet"]
+        brief: >
+          The [URI fragment](https://www.rfc-editor.org/rfc/rfc3986#section-3.5) component
+        examples: ["SemConv"]
       - id: full
         stability: stable
         type: string
@@ -23,19 +42,74 @@ groups:
           `url.full` SHOULD capture the absolute URL when it is available (or can be reconstructed)
           and SHOULD NOT be validated or modified except for sanitizing purposes.
         examples: ['https://www.foo.bar/search?q=OpenTelemetry#SemConv', '//localhost']
+      - id: original
+        type: string
+        brief: >
+          Unmodified original URL as seen in the event source.
+        note: >
+          In network monitoring, the observed URL may be a full URL, whereas in access logs, the URL is often
+          just represented as a path. This field is meant to represent the URL as it was observed, complete or not.
+          
+          `url.original` might contain credentials passed via URL in form of `https://username:password@www.example.com/`.
+          In such case password and username SHOULD NOT be redacted and attribute's value SHOULD remain the same.
+        examples: ["https://www.foo.bar/search?q=OpenTelemetry#SemConv", "search?q=OpenTelemetry"]
+      - id: password
+        type: string
+        examples: ["password"]
+        brief: >
+          Password of the request.
       - id: path
         stability: stable
         type: string
-        brief: 'The [URI path](https://www.rfc-editor.org/rfc/rfc3986#section-3.3) component'
-        examples: ['/search']
+        brief: >
+          The [URI path](https://www.rfc-editor.org/rfc/rfc3986#section-3.3) component
+        examples: ["/search"]
+      - id: port
+        type: int
+        brief: >
+          Port of the request
+        examples: [443]
       - id: query
         stability: stable
         type: string
-        brief: 'The [URI query](https://www.rfc-editor.org/rfc/rfc3986#section-3.4) component'
+        brief: >
+          The [URI query](https://www.rfc-editor.org/rfc/rfc3986#section-3.4) component
         examples: ["q=OpenTelemetry"]
-        note: Sensitive content provided in query string SHOULD be scrubbed when instrumentations can identify it.
-      - id: fragment
+        note: >
+          Sensitive content provided in query string SHOULD be scrubbed when instrumentations can identify it.
+      - id: registered_domain
+        type: string
+        brief: >
+          The highest registered url domain, stripped of the subdomain.
+        examples: ["example.com", "foo.co.uk"]
+        note: >
+          This value can be determined precisely with the [public suffix list](http://publicsuffix.org)
+      - id: scheme
         stability: stable
         type: string
-        brief: 'The [URI fragment](https://www.rfc-editor.org/rfc/rfc3986#section-3.5) component'
-        examples: ["SemConv"]
+        brief: >
+          The [URI scheme](https://www.rfc-editor.org/rfc/rfc3986#section-3.1) component identifying the used protocol.
+        examples: ["https", "ftp", "telnet"]
+      - id: subdomain
+        type: string
+        brief: >
+          The subdomain portion of a fully qualified domain name includes all of the names except the host name
+          under the registered_domain. In a partially qualified domain, or if the the qualification level of the
+          full name cannot be determined, subdomain contains all of the names below the registered domain.
+        examples: ["east", "sub2.sub1"]
+        note: >
+           The subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain,
+           such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+      - id: top_level_domain
+        type: string
+        brief: >
+          The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name.
+          For example, the top level domain for example.com is `com`.
+        examples: ["com", "co.uk"]
+        note: >
+           This value can be determined precisely with the [public suffix list](http://publicsuffix.org).
+      - id: username
+        type: string
+        brief: >
+          Username of the request.
+        examples: ["username"]