Introduce aliases for 7.x fields in 6.x #9283
Conversation
ruflin
added
in progress
|
Pinging @elastic/infrastructure |
|
@graphaelli I wonder if it would be helpful to start a similar PR against 6.x in apm-server to see if all the 1-1 alias do not already have an other field. If it does, CI will complain. |
|
Will these fields appear in 6.x documentation without descriptions? do we want that? There would be an option to hide them in docs if we are only adding them to not use them? |
|
@ruflin already in progress, I'll add you as reviewer |
|
I think I need to backport this one #9283 and probably one more for the docs to look properly. |
ruflin
force-pushed
the
introduce-aliases
branch
from
29c68ca
to
4cf3014
Compare
ruflin
added
review
and removed
in progress
metricbeat/docs/fields.asciidoc
Outdated
|
|
||
| -- | ||
|
|
||
| *`beat.hostname`*:: |
There was a problem hiding this comment.
I'm a little confused by the semantics here. Is beat.hostname the alias, or the field the alias will point to? Reading the above:
- name: host.hostname
type: alias
path: beat.hostname
makes me think that host.hostname is the alias that is going to point to the real field beat.hostname. I know the ECS field name is host.hostname. Is the plan in 6.x it'll be the alias, and in 7.x, it'll be the real field when the mapping support many : 1 (the 1 being the ECS field, the many being all variations that are used across all the beats).
Can you let me know the specific field that will require the many : 1 mapping? I thought I recalled Boaz saying something about filebeat, but looking at the fields (well at least the ones with host in them) I don't see where the issue specifically is. Just looks like all the beats have beat.hostname which would map to host.hostname, which should still be 1:1? Even though beat.hostname is a field in many different indices... ?
Sorry for so many questions, just trying to wrap my head around all this as I think about what should be tested!
There was a problem hiding this comment.
The fields that are aliased here are for 6.x and are only 1:1 mappings to make them compatible with 7.x data for Logging UI and Infrastructure UI.
For the many:1 mappings which are only introduce in 7.x you can find some examples here:
This form apache and nginx and you will see that these have many fields in common and the old fields are mapped to the same new fields.
There was a problem hiding this comment.
@stacey-gammon Here's my attempt at clarifying as well.
For 6.x and for 7.x, the aliases are flipping direction indeed.
6.x: new fields introduced as aliases to the old field they will eventually replace. This is for forward compatibility. We will start being able to depend on the new field names in the later releases in the 6.x line for those. This is only applicable for 1:1 mappings, however (explained below).
7.x: For all fields being renamed (1:1 or many to 1), the ECS field is now the canonical field holding the data, and old fields alias to the ECS field.
Many to 1 mappings: Filebeat modules are indeed where this is happening the most, but there may be other places. All modules used to nest their fields under a namespace [module name].[dataset].[field]. In many cases, different modules had perfectly equivalent fields: apache2.access.url and nginx.access.url. These two examples are becoming url.original. In 6.x we can't create a forward compatible alias to all of these values, as the alias can only point to one field. However in 7.x we'll have all of these old fields be aliases to the new canonical field. Nothing prevents having many alias fields pointing to the same field.
There was a problem hiding this comment.
This looks ok overall. Here's a few points, though:
- ECS doesn't have
cloud.project.id - Please review my question about
beat.name&agent.type. I don't think this is the right mapping.
Not a blocker for this PR, but we must address for 6.6:
- You mentioned in a comment that you'll need to backport the PRs to make the asciidoc for aliases to look good, but you pasted this PR's url instead of the correct PRs :-) The correct links are PR Fix alias field generation in docs #9269 and issue Improve fields view in Docs #9288. I've listed them both under the 6.x section in Transition Beats to ECS #8655.
| # Metricbeat fields | ||
| - to: event.module | ||
| from: metricset.module | ||
| index: metricbeat |
There was a problem hiding this comment.
I didn't realize we had both fileset.module and metricset.module becoming event.module. But I see you've treated them as two 1:1 mappings, in different indices. Good stuff.
|
Gil makes good points with some of the missing aliases:
Looking at the list of libbeat common fields, I've spotted a few more:
One we can't tackle because it's an object is |
|
@webmat
|
|
@graphaelli Ah you're right about |
|
ruflin
force-pushed
the
introduce-aliases
branch
from
230f221
to
fadb65f
Compare
dev-tools/ecs-migration.yml
Outdated
| # Beat fields | ||
| - from: beat.name | ||
| to: agent.type | ||
| alias6: true |
There was a problem hiding this comment.
Good find. Remove it and added event.timezone and container.id instead as it was missing.
|
Still looking good after the rebase, but I noticed the |
dev-tools/ecs-migration.yml
Outdated
| - from: beat.name | ||
| to: agent.type | ||
| alias6: true | ||
| alias: true |
There was a problem hiding this comment.
You're right. This whole section needed to go, not just the alias6 :-)
ruflin
force-pushed
the
introduce-aliases
branch
from
c07a437
to
f5adede
Compare
|
jenkins, test this |
|
Seems like the LS integration tests were flaky which I haven't seen before. Retrying. |
ruflin
force-pushed
the
introduce-aliases
branch
2 times, most recently
from
e61220a
to
dffc8c0
Compare
|
jenkins, test this |
This introduces for aliases for fields which will exist in 7.x and a 1-1 mapping is possible. Having this fields already around makes sure we do not use them otherwise. need at least 6.4 for testing trying out if snapshot trick works reset docker compose update more fields.go files rebase to get 6.4 environment remove not needed fields add migraton field update fields.yml again update fields rebase update migration document add alias docs
ruflin
force-pushed
the
introduce-aliases
branch
from
dffc8c0
to
8652806
Compare
|
The failures are very similar to before so I'm starting to guess it's related. Unfortunately so far I couldn't reproduce it locally. |
|
jenkins, test this |
|
Strangely now all tests except a new flaky one went green. Merged for now but will keep an eye on it. |
This introduces for aliases for fields which will exist in 7.x and a 1-1 mapping is possible. Having this fields already around makes sure we do not use them otherwise.
This introduces for aliases for fields which will exist in 7.x and a 1-1 mapping is possible. Having this fields already around makes sure we do not use them otherwise.