This issue is meant to collect breaking changes we want to do for ECS 8.0

[webmat]

Any suggestion for breaking changes should be submitted via their own github issue, not as a comment on this issue.

Decided

Schema

• Remove guidance to lowercase http.request.method #1363 Remove guidance to lowercase http.request.method
• Remove Go code generator #1567 Remove or relocate the Go library and the code generating it
  • Need to confirm if the library will remain a Beats dependency in future releases.
• RFC 0017 - remove log.original field #1580 RFC 0017 Remove log.original
• Remove host.user.* fields #1362 Remove host.user.*
• Remove process.ppid field #1596 Remove process.ppid, now that we have all of process.parent.* (which includes the equivalent field process.parent.pid)

Tooling

• Remove --oss flag from ECS generator script #1340 Remove --oss flag
• Remove use cases directory and code #632 Remove use-cases directory and associated code
• Remove Golang dependencies #630

Under consideration

Tooling

• [meta] Consolidate generated ES index templates #1587 Remove legacy templates as generated artifacts.

No longer under consideration for 8.0

Items previous listed as Under consideration but subsequently removed. May still have value in the future, so capturing below.

• http.response.status_code as long instead of keyword or integer #564 Do we need to change the type of http.response.status_code to keyword, in order to allow users to bucket HTTP traffic by status code?
• Adopting new version field type #887 [RFC] Version Type Field Migration - Stage 0 #1309 Introduce the version data type for version fields
• [RFC] Label fields for additional types - Stage 0 #1341 Add other types of labels, potentially nested within labels.*, e.g. labels.long.*
• Does migrating @timestamp from date to date_nanos require a breaking change, or are the types compatible? Support for nanosecond timestamps (date_nanos) #1065
  • The two types are compatible, but with 64-bit integers not supported in Javascript, Kibana has several outstanding issues (64 bit number/integer support in Kibana kibana#40183, Canvas needs to support date_nanos kibana#41441, [Logs UI] The UI throws error when @timestamp format is date_nanos kibana#88290) that would create a degraded experience if data_nanos was the default for @timestamp, I think.
  • May still be able to provide guidance for those ECS users who require the additional precision of date_nanos and understand the possible side-effects.
• Update related.* field names from singular to plural #909 Update singular names to plural in the related.* field set.
  • The cost of introducing this as a breaking change outweighs the inconsistency in the naming. Not pursuing at this time.
• Fix the timestamps on file.* to be less Linux-centric
  • I see why the file timestamp fields could introduce confusion, but I also think with the new usage documentation sections we might be able to provide better guidance without changing the field names. Other tools/languages/libraries also use mtime, ctime, and atime to report on timestamps, even across different filesystems and OS types.
• Having category fields that don't assume a document is a time-based event
  • Happy to revisit this item, but I'm not finding any past context. Also not aware of any recent use cases.
  • We want to continue expanding ECS to better support full-text search use cases (searchable corpora), but if we need to add categories or other event kinds/categories/types, I don't see those as breaking changes.
• Adjust case normalization in accordance with upcoming query time case sensitivity
  • For some use cases, the original case of a value is something we want to preserve. Some short keyword fields, like protocol names, may remain good candidates for normalizing to all lowercase.
  • The case_insensitive parameter is available on some query types since 7.10. This provides an option to case-insensitivty matching without worrying about the original value's casing in ECS.
  • Could extend the normalize parameter introduced in Mark which fields should be arrays. #727 to identify lowercase fields in the schema docs.
  • Remove "implementing ECS" reference from field descriptions #1364 Remove "implementing ECS" pointer from normalized fields' descriptions.
• Clarify guidance on OS detail fields so they're populated more consistently across sources. (related Operating System field for Linux, Windows, macOS #576)
• Add meta information to fields where this applied #721 Where could the emerging standard metadata entries fit into ECS? Document standard metadata entries. elasticsearch#61941
• Deprecate host.hostname to avoid confusion between host.hostname and host.name
  https://github.com/elastic/observability-dev/issues/1056
  • Host identifiers RFC
• User.hash vs. new hash.* fields #483 Reuse hash.* underneath user.* in place of user.hash?

[rw-access]

There was also talk about embedding file in more places, since several fieldsets use .file and .path. The names we get as a result are a little confusing. Maybe we can find better ways to accomplish reusability

The first ones that I remember:

• process.file.*
• dll.file.*
• driver.file.* (haven't created driver yet)

[ebeahan]

#838 Remove guidance to lowercase http.request.method

This is going in for 1.6, right?

[webmat]

@ebeahan Yes it is

[djptek]

Do we need to change the type of http.response.status_code to keyword, in order to allow users to bucket HTTP traffic by status code? See #564

Prior to Lens, this might have been accomplished using e.g. ranges to bucket the 2xx, 3xx, 4xx, 5xx codes

This has been improved and simplified with Lens, which (tested 7.11.2) supports Break down by and other bucket strategies vs long out of the box, for example this one gives a full breakdown of status codes

Note: to get a count per value in Lens you may need to increase Number of Values in Break down by configuration. The max value of 100 is sufficient to exceed the 41 response codes listed in https://tools.ietf.org/html/rfc7231#section-6.1

Here is a data set if you'd like to try that, the field will map to long using dynamic mapping

POST test/_bulk
{ "index" : { "_id" : "1" } }
{ "http.response.status_code" : 200 }
{ "index" : { "_id" : "2" } }
{ "http.response.status_code" : 200 }
{ "index" : { "_id" : "3" } }
{ "http.response.status_code" : 200 }
{ "index" : { "_id" : "4" } }
{ "http.response.status_code" : 201 }
{ "index" : { "_id" : "5" } }
{ "http.response.status_code" : 201 }
{ "index" : { "_id" : "6" } }
{ "http.response.status_code" : 202 }
{ "index" : { "_id" : "7" } }
{ "http.response.status_code" : 300 }
{ "index" : { "_id" : "8" } }
{ "http.response.status_code" : 301 }
{ "index" : { "_id" : "9"} }
{ "http.response.status_code" : 404 }
{ "index" : { "_id" : "10"} }
{ "http.response.status_code" : 404 }

[ebeahan]

@jamiehynds I believe you added this item?

Consider removal of host.os.full.caseless/text fields and host.os.name.caseless/text fields. Example where these fields can cause unnecessary duplication.

The *.caseless fields are actually not specified in ECS. I believe it's a custom multi-field added in the endpoint schema: https://github.com/elastic/endpoint-package/blob/master/custom_schemas/custom_os.yml

I'm removing it from consideration for now. Happy to re-add if I'm misunderstanding.

[jamiehynds]

Good catch - thanks @ebeahan. I'll discuss further with the endpoint team, but fine to remove from our list of proposed breaking changes.

[ebeahan]

A release branch has been cut for ECS 8.0, and the information in the changelog supersedes what's included in this issue.