Importing saved objects can break "weak links"

[robcowart]

Kibana version:

8.0.0-rc1

Elasticsearch version:

8.0.0-rc1

Server OS version:

Docker container

Describe the bug:

When importing into 8.0.0-rc1 saved objects which were exported from from 7.x (tested with 7.14.0 and 7.16.3 exports) the UUIDs of dashboards are not preserved. The import was via the Kibana UI (import via API was not tested). "Create new objects with random IDs" was NOT selected, rather the default "Automatically overwrite conflicts".

Dashboard UUIDs are frequently used in markdown visualizations to provide navigation options, allowing users to easily move in-context from one dashboard to another. In our case this will result in 100s of hyperlinks which will have to be manually edited after import.

This will affect all users that will move to 8.0.0 by migrating to a new cluster rather than upgrading in place.

Steps to reproduce:

1. Export a dashboard from 7.16.3 (one is attached) and import it into 8.0.0-rc1, ensuring that "Create new objects with random IDs" is NOT selected.
2. Open the dashboard and check its UUID in the URL path.
3. Delete the dashboard and reimport.
4. Again open the dashboard and check its UUID. It will be different.

Expected behavior:

As was the case in 7.x, the UUID specified in the import file should be preserved.

Any additional context:

export_7.16.3.ndjson.zip

[robcowart]

Another use-case impacted by this issue is when importing ML jobs which have custom URLs defined to launch Kibana dashboards. If the dashboard UUIDs change it will break the ML custom URLs.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[robcowart]

I am not sure why this has been associated with Team:Security. This has nothing to do with the Security/SIEM app, nor with RBAC. It is an issue of importing saved objects.

[pgayvallet]

I am not sure why this has been associated with Team:Security

That's not a mistake. The savedObjects spaces feature is owned by kibana-security, and the whole id rewritting behavior is a consequence of the shareable saved objects feature, that is associated to spaces.

[robcowart]

Thanks for the clarification

[rudolf]

@robcowart I understand that the association isn't obvious. The platform security team owns spaces and as part of a feature to allow sharing to multiple spaces saved object id's might be rewritten when upgrading to 8.0 #27004 #100489

Although the id's changed we made every effort to avoid it affecting e.g. deep links, but I'm hoping we can share public documentation with you as a better reference for understanding it.

@jportner #100489 is marked as a breaking change but it doesn't show up in the breaking changes release notes section. I think it would help if users understand the impact of the changed id's.

[robcowart]

I read through those issues and it is unclear how an upgrade from 7.x to 8.x will affect the existing saved object UUIDs. For example... in 7.17.x I have Space A and Space B, both of which have saved object X. I then upgrade to 8.0. Will X remain X in both A and B, or will it be changed to Y and Z respectively?

[jportner]

@jportner #100489 is marked as a breaking change but it doesn't show up in the breaking changes release notes section. I think it would help if users understand the impact of the changed id's.

Good point, I think we can put something together for the release notes.

I read through those issues and it is unclear how an upgrade from 7.x to 8.x will affect the existing saved object UUIDs. For example... in 7.17.x I have Space A and Space B, both of which have saved object X. I then upgrade to 8.0. Will X remain X in both A and B, or will it be changed to Y and Z respectively?

Sorry for the confusion here, it involves a lot of implementation details. We made the change because we really needed to have globally unique saved object IDs to enable shareable saved objects. Basically:

When you upgrade Kibana to 8.0:

Any old saved objects that exist in a custom space will have their IDs changed. This ensures that every saved object ID is unique. For example: before, you could have dashboard "123" in the Default space and dashboard "123" in Another space, but after the upgrade you would have dashboard "123" in the Default space and dashboard "456" in Another space.

Existing URLs are preserved with special "legacy URL alias" saved objects and should continue to function seamlessly. In addition, each saved object's "origin" is preserved for import/copy purposes.

After you upgrade Kibana to 8.0:

It doesn't matter if your saved object was exported in 7.x or 8.0, Kibana takes the following steps during import/copy:

1. If Kibana finds a matching saved object with the exact same ID, that will be the import destination -- you can overwrite that destination, or skip it.
2. Otherwise, if Kibana finds a matching saved object with a different ID that has the same "origin", that will be the import destination -- again, you can overwrite that destination, or skip it.
3. Otherwise, if a saved object with the exact same ID exists in a different space, then Kibana will generate a random ID for the import destination.
   • If you are using a new cluster as described above, basically: the first time you import a given saved object into Kibana, it will retain its ID (see step 4 below). The next time you import it (into a different space), it will be caught during this step and it will be given a random ID.
4. Otherwise, Kibana creates the saved object with the given ID.

There are a few corner cases, but that captures the main scenarios. You appear to be experiencing scenario (3).
There's no indication in the import UI itself that the saved object ID has changed, but the API response does inform you that the saved object has a different destination:

{
    "successCount": 1,
    "success": true,
    "warnings": [],
    "successResults": [
        {
            "type": "map",
            "id": "2c9c1f60-1909-11e9-919b-ffe5949a18d2",
            "meta": {
                "title": "[eCommerce] Orders by Country",
                "icon": "gisApp"
            },
            "destinationId": "9272c990-98ee-4614-a11a-ffe4ca77ce16"
        }
    ]
}

Dashboard UUIDs are frequently used in markdown visualizations to provide navigation options, allowing users to easily move in-context from one dashboard to another. In our case this will result in 100s of hyperlinks which will have to be manually edited after import.

As I mentioned above, and it seems like you discovered, this will only affect new clusters. Any time Kibana is upgraded with existing data, existing URLs are preserved and this won't be an issue.

I apologize this change has negatively impacted your use case. I'm struggling to think of a technical solution that we could put in place to prevent this, but the truth is these markdown URLs are basically "weak links" to other saved objects, and when the saved object IDs change, they will simply break, and I'm not sure we have a good way to maintain them.

Potential solution 1

We could make these changes:

1. Pass in more data to the import hooks, a map of all saved object IDs -- what they were in the imported NDJSON file mapped to what their destination IDs became
2. Add an import hook for the visualization saved object type, which only executes when it is a markdown visualization, to essentially do a dumb find+replace for saved object IDs in the markdown content

One downside of this is that it only works if all "weakly linked" saved objects are present at import time. Another downside is that it could have unintended consequences if the saved object ID was not a random UUID (for example if the ID was simply "dashboard"...)

Potential solution 2

We could add an import hook for the visualization saved object type which only executes when it is a markdown visualization, which uses RegEx to find certain local URLs (just dashboards? ex: /app/dashboards#/view/123?...) and searches for the corresponding saved object; if the saved object is not found, it can fall back to search for a saved object with the matching origin, and in that case it could update the markdown accordingly.

This would be more resilient but it wouldn't be trivial to implement.

I think what I struggle with here is that I'm not sure how common this use case is. Do lots of users have visualizations with these "weak links"? If so, it might be worth our time to implement this solution. @elastic/kibana-vis-editors can you weigh in here?

---

Workaround

@robcowart I think you can sidestep this issue for now by regenerating the object IDs before you import them. You could produce a small script that does the following:

1. Reads an NDJSON file and enumerates all saved object IDs
2. Generates a new random ID for each saved object
3. Executes a find+replace for each ID in the entire file (which should affect the saved object ID and the "weak link" in the markdown)

Of course this would only work if all of your dashboards/etc are present in the same file.

Edit: I took a look at the ElastiFlow docs and, while it looks like you've got all saved objects present in a single file, you're providing that file for users to download, so this script-based workaround won't really be user-friendly.

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

The markdown "visualization" is not owned by the VisEditors team (it's a presentation team thing). But TSVB has several places which allows links like this as well which are affected as well, so let's keep it like this.

I feel like I'm missing a key part here, but shouldn't the legacy URL aliases be written on import as well? I was under the impression importing old saved objects would be just like upgrading with them in the kibana index. If this is not the case, all the things that are running on legacy URL aliases will break in case people move saved objects over the 8.0 boundary via export/import: reporting URL in cron job scripts, bookmarks in people's browsers and so on. We can't possibly fix them within the system, so we should come up with a solution that works for these like it works for Kibana upgrade.

About the potential solutions in #123550 (comment)

This is a core team topic, but right now it's not possible to "look up" other saved objects during migration - it's a synchronous API on purpose. This came up a few times, but allowing to make requests during the migration causes a bunch of issues and can easily become a serious performance bottleneck.

I'm also not confident some kind of regex to find the right URLs will work reliably - there are so many ways to specify a URL to a dashboard on a Kibana cluster and in TSVB it's even possible to have dynamic data as part of these (using the {{}} syntax).

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[robcowart]

Hmmmm... I fear this is going to be very problematic. We are going to have to do a ton of testing to make sure we cover all scenarios, and probably re-build all of our content (350+ visualizations, 50+ dashboards, 120+ ML jobs, and 140+ Alerts) to allow all scenarios to co-exist in a single 8.0.0 cluster. This is going to be weeks of unplanned work.

What is the timing for 8.0.0 GA? We need to figure out if we can make changes ahead of time, or whether we will have to send a notice out to our 100+ customers and 30,000+ users that they need hold off on 8.0.0 until we give the go ahead.

[robcowart]

Another question. Reading the above it doesn't sound like it will be possible to have a "dev" space and "prod" space. A very common scenario when making changes to "prod" is to...

1. export everything from "prod" and import it into a "dev" space.
2. make changes in the "dev" space.
3. export from "dev" and re-import into "prod".

It sounds like step 2 would cause the UUIDs to change, which would then no longer match for step 3 to update the existing objects. Is this correct?

[jportner]

I feel like I'm missing a key part here, but shouldn't the legacy URL aliases be written on import as well? I was under the impression importing old saved objects would be just like upgrading with them in the kibana index.

Apologies in advance if I gloss over some details, I am attempting to summarize technical discussions and decisions that took place over ~15 months and a dozen PRs. I'll just call them "aliases" for short.

We explicitly made a decision not to attempt to create aliases on import. These aliases exist to make sure that bookmarked URLs still work after Kibana is upgraded. That's their only purpose, we don't want to proliferate aliases because we eventually want to get rid of them -- we consider aliases to be technical debt that we were forced to shoulder to get to a state where we can have globally unique object IDs.

Aliases are a band-aid on the problem of changed object IDs, not a long-term solution. They are a burden on consumers who had to implement extra UI logic to handle the different outcomes (exact match, alias match, conflict...), and they reduce performance on the server due to an extra round-trip to Elasticsearch every time a user wants to fetch an object using its alias.

---

Implementation details...

There are two types of saved object migrations:

1. Consumer-defined migrations (defined by consumers in the type registration)
2. Conversions and reference transformations (defined by Core) -- this is what regenerates the saved object ID, creates a legacy URL alias, ensures that outbound references are still linked correctly, etc.

Migrations can happen in two situations: (a) during upgrade, and (b) at runtime when objects are created/imported.

Importing an old saved object into Kibana executes the consumer-defined migrations, but not conversions and reference transformations. The conversion process only happens when Kibana is upgraded. The main reason for this is that the migrations are the last thing that happen in a long chain of events when a saved object is created. For example:

1. Import (executes several searches via find before finally bulkCreate)
2. Spaces saved objects client wrapper
3. Security saved objects client wrapper
4. Encrypted saved objects client wrapper
5. Repository (executes preflight checks, then applies migrations, then creates raw document in Elasticsearch)

The very last thing that happens before an object is created is that the migrations are executed. We explicitly skip the conversion process at runtime because 1. it proliferates legacy URL aliases, which I mentioned we want to avoid, and 2. it would create way too many corner cases and drastically increase complexity.

Prior to 8.0, Kibana is in a known state where "isolated" objects each exist in only one space, and all objects are being changed at the same time. During the migration process we regenerate object IDs (and outbound references) deterministically based on the saved object's ID and its current space.

Let's assume for a moment that we wanted to also execute conversions at runtime (when an object is imported). What if the destination object has been shared to multiple spaces, or all spaces? What if an object with the same destination ID already exists, but in a different space? What if some of the alias(es) have been disabled? What if any of the outbound references had been changed during import (for example, in the case of a missing reference to a data view)? In all of these cases, conversions that happen transparently at the very last step before an object is created poses a huge problem. On top of all that, consumers wouldn't be aware what ID the object was actually created with.

This is why we added extra logic to the import process to handle origin matches, etc, and change IDs if necessary during import, before attempting to create/re-created any saved objects.

[jportner]

Whoops, I commented too soon!

If this is not the case, all the things that are running on legacy URL aliases will break in case people move saved objects over the 8.0 boundary via export/import: reporting URL in cron job scripts, bookmarks in people's browsers and so on. We can't possibly fix them within the system, so we should come up with a solution that works for these like it works for Kibana upgrade.

You're talking about folks who may export all of their objects and import them into a new cluster? If they are importing the same objects into multiple different spaces, then yes, what you're describing would happen.

I think the friction here is that exports/imports happen within a single space, and an object's space-related metadata is stripped when it is exported. If we wanted to support "export everything from a prod cluster and import it into a dev cluster", then that would be better served by an export-across-multiple-spaces feature, which we have an open ER for here: #91615

The current export/import behavior allows users to create "pseudo-copies" of objects, e.g., creating at-most-one-copy of an object in a given space. This was an unintended side effect when we originally implemented Spaces, and while it can be useful in some cases, we don't want users to rely on this behavior. It took a lot of work just to maintain this behavior when we changed how saved objects are serialized to Elasticsearch for share-capable objects (which is why we needed to start enforcing globally unique object IDs).

[jportner]

Another question. Reading the above it doesn't sound like it will be possible to have a "dev" space and "prod" space. A very common scenario when making changes to "prod" is to...

1. export everything from "prod" and import it into a "dev" space.
2. make changes in the "dev" space.
3. export from "dev" and re-import into "prod".

It sounds like step 2 would cause the UUIDs to change, which would then no longer match for step 3 to update the existing objects. Is this correct?

If you're talking about two spaces in the same instance:

Step 1 would cause the object IDs to change in the "dev" space. When the objects are exported from "dev" and re-imported into "prod", the objects would be matched with their origins -- so the object IDs would change back, but any weak links that were created in the "dev" space" would be broken.

If you used two different clusters to do this (a "prod" cluster and a "dev" cluster), the object IDs would not change during import, and weak links would be maintained.

[robcowart]

Sorry. I meant step 1 would cause the UUIDs to change... but you understood what I meant. That is a bit disappointing that it will work like that. While I am only aware of a few organizations that have the luxury of a dev cluster. I know a lot that have one or more dev spaces.

To simplify this for our users, we might just have to build our own Elastic Stack exporter/importer that takes all of the dependencies into account and allows us to work around these new limitations.

[jportner]

Sorry. I meant step 1 would cause the UUIDs to change... but you understood what I meant. That is a bit disappointing that it will work like that. While I am only aware of a few organizations that have the luxury of a dev cluster. I know a lot that have one or more dev spaces.

To simplify this for our users, we might just have to build our own Elastic Stack exporter/importer that takes all of the dependencies into account and allows us to work around these new limitations.

I don't think that will be necessary!

Apologies for any heartburn this may have caused, and thanks for bringing this use case to our attention. While this is currently behaving "as intended", I believe we have overlooked the impact that these changes have for weak links between saved objects.

We met today and we're planning to implement an enhancement to the import and copy-to-space features to give users the option to preserve URLs. I think this will satisfy your use case and ensure that users can seamlessly upgrade.

We are tracking this as a "known issue" for the 8.0.0 release via #124987, and we hope to ship a fix with the 8.0.1 / 8.1 releases.

[robcowart]

Thanks, that is good to hear. If it helps I can provide examples of where we currently use such links... this includes Markdown (both Markdown visualization and TSVB Markdown), ML custom URLs, and as a default page in Kibana advanced settings.

[jportner]

I opened two separate issues that are prerequisites to fixing this bug:

1. Enhance legacy URL aliases for broader usage #127047
2. Prevent multiple objects of the same origin in a space #127054

Once those are complete, we can implement a new feature that adds an option in the Import and Copy UIs to "Preserve weak links" (exact phrasing TBD). That final task will close out this bug.

[drewdaemon]

It looks like the prerequisite tasks are done here. Do we have a plan for addressing this? It continues to be a pain point.

[robcowart]

I can confirm that it is still an issue. The most disappointing part of it is that the functionality brought by the changes has added no value for any of the organizations with which we work. So it has overall been a net loss of usability.

Perhaps what is needed is some type of scope that can be applied to saved objects, where...

• An object scoped as single-space can never be shared between spaces (the old behavior), so it doesn't matter if it has the same ID as another single-space object in another space.
• An object scoped as multi-space can be shared with other spaces (the new functionality) and must have an ID that is unique across all spaces.

This would allow both the legacy and the new behavior to co-exist without having to try to perform magic with URLs.

[pgayvallet]

@elastic/kibana-security unassigning Joe for obvious reasons. Is anyone able to follow-up on that one?

Looking at #123550 (comment), it seems the prerequisites are all done, but I'm not sure how you were seeing the next step being done:

Once those are complete, we can implement a new feature that adds an option in the Import and Copy UIs to "Preserve weak links"

[flash1293]

If I remember correctly the idea was to also create the legacy alias objects along with the saved objects so the behavior is identical for copy-with-collisions and upgrade-with-collisions

[mholttech]

This is impacting our ability to load OOB dashboards installed by Fleet Integrations to additional Spaces causing us significant issues in onboarding new customers in my company. Release Notes for 8.0.0 stated that this would be fixed in 8.0.1/8.1.0 but here we are at 8.4.1 and this continues to persist. Can we get an ETA on when this BUG will be fixed?

[azasypkin]

UPDATE: Kibana Platform Security team met with the Kibana Core team yesterday to discuss this issue. We preliminary agreed on the potential solution, and Kibana Platform Security team is going to pick this up in the coming weeks.

Can we get an ETA on when this BUG will be fixed?

We cannot share ETA, unfortunately, but we'll make sure the progress is visible.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[robcowart]

@azasypkin can you share a bit about the proposed solution? It would be good to get an idea of what the outcome will be and have the opportunity to provide feedback. Thanks.

[azasypkin]

@azasypkin can you share a bit about the proposed solution? It would be good to get an idea of what the outcome will be and have the opportunity to provide feedback. Thanks.

Sure, the preliminary idea is to introduce a new flag in the import UI and API that would generate legacy URL aliases for all imported objects that have to change IDs during import (like mentioned in #123550 (comment) and #123550 (comment)). This flag will be "unchecked" by default.

At this point, it sounds like the most reliable option that also comes with the reasonably low risk.

[seang-es]

Will this also cover cases where saved objects like Dashboards are copied from one space into another? We're seeing broken links in the navigation panels in that case as well, and want to be sure the links can be rewritten to match the new object UUIDs.

[azasypkin]

Will this also cover cases where saved objects like Dashboards are copied from one space into another?

Presumably, yes, we should cover copy case as well. I'll confirm here once we pick it up.

[lachlann562]

We are new to using spaces, but the use case described of having a dev space vs a production space with fleet is exactly the use case we encountered.

Example:
We used fleet to setup numerous default dashboards for metricbeat. (e.g. System Overview, Host Overview)
We want to copy these dashboards to various spaces for different teams which restricts access to WHICH metrics they can access. (ie which servers they can use these dashboards for)
After copying, all of the markdown and automatic drilldown features are broken because they all reference the dashboards in the "default" space. (System Overview cannot drilldown to Host overview, and the markdown in Host Overview does not refer back to the System Overview in the new space).

[nicpenning]

Encountering this issue in 8.6.2 today and was a big let down.

When we copy OOB integrations Assets to another space, the links break because new IDs are generated.

Please create a "Share to Spaces" feature like that of the Data View Object.

Let me know if anyone needs more info or the use case.

We are attempting to use the System Windows Security Dashboards that has the links to other dashboards to navigate to but the IDs never align after copying to so we either have to remove that visualization or update every link with the appropriate ID and do this for every space we import into.

[azasypkin]

Thanks for sharing your use case @nicpenning,

When we copy OOB integrations Assets to another space, the links break because new IDs are generated.

Do you copy via UI or API, by the way?

We are attempting to use the System Windows Security Dashboards that has the links to other dashboards to navigate to but the IDs never align after copying to so we either have to remove that visualization or update every link with the appropriate ID and do this for every space we import into.

We're working on a short-term solution here #149021. But since linking dashboards through Markdown visualizations is basically a fragile workaround for something that Kibana currently doesn't provide, the long-term solution would be a first-class support for "Navigation Embeddables" discussed here #125313

[nicpenning]

No problem! This was done all through the UI. I didn't try the API.

I will take a look at those two issues. Thanks!

[nicpenning]

The navigation embeddable is a great idea and I would enjoy seeing that implemented.

I am thinking the true feature to reduce dashboard version fragmentation is the ability to share a single dashboard across any amount of spaces. Do you think that should be an entire issue/FR altogether?

To me that would prevent any need for importing/ copying all around and when a dashboard gets updated you don't have to go through all of that work again.

[azasypkin]

I am thinking the true feature to reduce dashboard version fragmentation is the ability to share a single dashboard across any amount of spaces. Do you think that should be an entire issue/FR altogether?

Yep, I agree. You can follow the progress on this and/or share your feedback in #127795. The priority (and future) of this feature isn't clear yet though.