Add saved object tags to integrations with Kibana saved objects

[alexfrancoeur]

The Kibana team (well, @pgayvallet 😉) has a PR for saved object tagging targeted for 7.11 that is currently under review (elastic/kibana#79096). This is the MVP and we plan to iterate. That being said, I think there is an opportunity to update any packages with Kibana saved object tags in the same release. I realize the registry is not release based, but it'd be a nice announcement to go along with user generated tags 😄

Next up that's related to this effort, I plan to start up discussions with the team on elastic/kibana#70461

cc: @ruflin @mostlyjason @tbragin @kobelb @joshdover

[ruflin]

Mentioning @andresrc as he will drive the updates of integrations. I assume it means, any visualization / dashboard etc. where we will start using tags, will be only compatible with KB 7.11, means the package itself is probably only compatible with 7.11. No problem with that, just wanted to make sure compatibility is clear.

[tbragin]

@alexfrancoeur this is great news! +1 on automatically applying tags for the dashboards we ship with integrations. Thank you for starting this conversation ❤️

cc @sorantis @mukeshelastic from observability integrations
cc @jamiehynds @hmnichols from security integrations

[alexfrancoeur]

As a quick update to this group, @pgayvallet has officially merged tags with out of the box support for dashboards and visualizations. We'll be discussing what additional improvements we can make for 7.11 (elastic/kibana#81980), but the functionality is now available.

[ruflin]

@andresrc We should start to create a guideline on how we use the tags inside the packages

[bradenlpreston]

@jamiehynds and @epixa in the event we want to update any packages related to security.

[nreese]

Tags added to maps with elastic/kibana#83197

[alexfrancoeur]

Hey all, I wanted to provide a quick update around tags for 7.11.

There has been some significant (and amazing) progress from @pgayvallet, @ryankeairns and team to support a global experience for tag usage.

We now have search syntax to filter down results by type and tag elastic/kibana#83422
Tags are now a suggested filter from our navigational search elastic/kibana#85144
To support this even more, we'll soon surface tags in search results with saved objects that have them elastic/kibana#85084
Users can now manage tags in bulk across supported saved objects elastic/kibana#84177

This significantly increases discoverability and streamlines navigation of Kibana assets packaged with integrations.

Have we come to a consensus on guidelines for integrations? If so, do we think that integrations will add support for tags in 7.11? I also believe that integrations are not bound by release, so maybe it's possible to iterate here.

[tbragin]

@mukeshelastic @sorantis @andresrc @mostlyjason @jamiehynds @ruflin @ph I'd love your perspective here.

As you may recall, we asked the Kibana team to prioritize tags, so we have a mechanism to organize assets we are installing with modules and integrations. Do we already have a plan in place to leverage them? I know it's been discussed, but not sure where we ended up.

[ruflin]

My suggestion is we start simple with 2 tags:

• integrations name tag on all assets from an integration
• managed tag on all the assets to indicated that they are managed by us. Alternative we could add a fleet tag to indicate that they are managed by fleet.

[sorantis]

We discussed the possibility to use tags as a mechanism to automatically expose integrations' assets in the UI. For example we could automatically add a link to a dashboard based on a tag matching service/integration name on a host's or container's details page. This actually aligns with @ruflin's first suggestion.

[ph]

@ruflin what you are proposing is to thave theses tags hardcoded at install time? The second steps would be to allow dev author to define more tags?

[ruflin]

Not sure what you mean by hardcoded. EPM would take the package name, create a tag out of it and add it to all assets. In addition, it adds the managed tag to all assets.

For your second part about devs adding tags: I'm wondering if a tag references the saved objects or if a tag is part of each saved object. If it part of each object, I guess it could already be used today? @alexfrancoeur might know more here.

[pgayvallet]

For your second part about devs adding tags: I'm wondering if a tag references the saved objects or if a tag is part of each saved object

The reference is from the saved object referencing the tag to the tag SO

[ph]

Not sure what you mean by hardcoded. EPM would take the package name, create a tag out of it and add it to all assets. In addition, it adds the managed tag to all assets.

Yes this exactly what I wanted to hear, this really make it simple.

@pgayvallet so this is a N:N relationship and not denormalized.

In that case maybe it's not a good idea to allow integration author to add tags, I would be worried about tag explosion. But maybe something that @masci can investigate?

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[alexfrancoeur]

Apologies to keep pinging on this one, but do we have a sense of target release or priority to start supporting tags with integrations? I see this was moved to backlog and do not see any PR's linked. One of the initial drivers to introduce this feature was to organize Kibana assets in packages. It feels like the technical implementation is relatively simple, it's just a matter of defining and agreeing upon a process. There is potentially still an opportunity to release alongside 7.11.

[masci]

@alexfrancoeur this is in the backlog as it still needs to be scheduled, we won't deliver anything in 7.11. As this will be a cross-team effort also on our side, I'll outline the proposed changes (most likely what ruflin said) along with an action plan during 7.12.

[alexfrancoeur]

Awesome, thank you so much for the update @masci !

[masci]

We discussed the issue and my team recommendation is to start by letting EPM add the managed tag at install time and postpone any manual tagging on the package side until we have a better idea of the integrations developer experience and how those tags could be leveraged throughout the user interface.

[tbragin]

@mukeshelastic @alvarolobato FYI ^^

[alvarolobato]

@masci does this mean we wouldn't be adding a tag with the package name or some way to identify the integration?
One of the main use cases I thought for this feature is precisely what @sorantis commented here. Right now when you are in Metrics UI you can only see so many details about the services running in a given host and there is no difference if you have a package installed for that service or not, we would like to link to the assets a given module installed and allow the users, for instance, to navigate to MySQL dashboards from a host that is running that service. This would provide a connection between the two worlds, packages with assets and the curated UI.

[masci]

@alvarolobato on a second read, the example in my comment might have been misleading.

The idea is to start by letting EPM add tags automatically based on whatever informations the EPM itself has at install time: managed is an example but also the name of the package is a good candidate for automatic tagging.

We are totally on board with the idea you described in the comment but what we propose not to do during this first phase is having developers adding tags before we know exactly how these tags would be used in the UI, a formal naming convention to avoid duplicates or similar variations of the same tag, and ultimately a better idea of how the granularity concept will affect a package containing multiple integrations.

So the plan would be:

• phase one: only automatic tags added at install time
• phase two: integrations developers add tags beforehand, within the package

@exekias please correct me if I'm wrong

[alvarolobato]

That makes sense to me, thanks for clarifying Massi.

…

On Sun, Mar 28, 2021, 14:36 Massimiliano Pippi ***@***.***> wrote: @alvarolobato <https://github.com/alvarolobato> on a second read, the example in my comment might have been misleading. The idea is to start by letting EPM add tags automatically based on whatever informations the EPM itself has at install time: managed is an example but also the name of the package is a good candidate for automatic tagging. We are totally on board with the idea you described in the comment but what we propose *not* to do during this first phase is having developers adding tags before we know exactly how these tags would be used in the UI, a formal naming convention to avoid duplicates or similar variations of the same tag, and ultimately a better idea of how the granularity concept will affect a package containing multiple integrations. So the plan would be: - phase one: only automatic tags added at install time - phase two: integrations developers add tags beforehand, within the package @exekias <https://github.com/exekias> please correct me if I'm wrong — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#327 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEC42BHK5OVQLL3XYEHCNT3TF4WG5ANCNFSM4SY3V2ZQ> .

[exekias]

++ let's make sure we get the Integration name automatically in, that will give us a lot already

[mukeshelastic]

+1 to adding tag representing integration name when user installs integration. This will allow users to filter dashboards by integration name as well solutions - metrics or APM provide option to surface links to dashboards with appropriate filters.

[sorantis]

@masci for packages we're also maintaining a list of categories. Is this something we can also consider in phase one?

[masci]

@sorantis that would imply we add the info in the package manually - while that is a good example of a formal label we'll certainly add in the future, I'd wait for the granularity example (specifically, AWS) to be ready before going through the packages we have and add the label.

[joshdover]

A few notes about a few things that should be considered when implementing this:

• Tags are their own saved object type, each idenified with a UUID like all other SO types.
  • This means integrations will need to include both a new tag object for each tag they want to add (eg. "system" or "managed") as well as adding a new object reference to each object that should have that tag.
• Any tags that are intended to be shared across integrations need to use the same UUID
  • For example, if we want include a "managed" or "fleet" tag on objects from all integrations, we need to make sure the same UUID is used for each integration. If a different UUID is used, then a duplicate tag will be created with the same name. This sounds like something that would confuse users.
  • I'm curious if we have any thing in place for sharing SOs across integrations so that we can be sure the same SO UUID is referenced for integrations that need to make use of it. I imagine we're doing something similar for the logs-* and metrics-* index patterns that we may be able to reuse here.
• We need to add support for installing SO of the tag type on the Fleet side. I've opened an issue to track this here: [Fleet] Add support for installing tag saved objects kibana#110660

[joshdover]

@masci Do you think this will still make 7.16 on the integrations side? We have a PR ready on the Kibana side but we haven't yet added this to the package-spec. elastic/kibana#114110

[mtojek]

Regarding the spec changes, we have added it recently as we exported dashboard with a tag: spec code

[alexfrancoeur]

Hey all, I wanted to follow up on this. Did we end up getting tags as part of the integrations package in 7.last? I couldn't find any linked issues

[joshdover]

I've opened an issue to add a standard set of tags to all Saved Objects installed by Fleet: elastic/kibana#123904

[alexfrancoeur]

@jen-huang I think this can be closed now. Correct?

[jen-huang]

Yep, done with elastic/kibana#123904 in 8.4.