Lens on Dashboard 7.12.1 Smoke Tests

[ThomThomson]

Summary

part of #102416

This PR adds dashboard migration smoke tests for a variety of lens panels. These will ensure that every time a migration issue is introduced that would break Lens panels on dashboards, it is caught by the CI.

These smoke tests are written in such a way that they would have caught #101430 and #100756 before the 7.13 release.

It works by importing a dashboard last saved in 7.12.1 which contain 6 lens panels in different configurations, with and without drilldowns, by value and by reference. It then checks that all panels load correctly, that there are no error embeddables, that each panel can be edited when in edit mode, and that all drilldowns were retained.

How to test

Preventing severe errors during migration:

You can simulate #101430 by adding the following if statement after line 29 in src/plugins/embeddable/common/lib/migrate.ts

if (!enhancementDefinition.migrations[version]) {
  throw new Error('Simulated error while migrating!');
}

With this simulated error, the smoke test should fail when importing the dashboard saved object, as by value embeddables with drilldowns were included in the smoke test dashboard.

Preventing severe runtime errors after migration:

You can simulate #100756 by adding the following if statement after line 1210 in x-pack/plugins/lens/public/indexpattern_datasource/operations/layer_helpers.ts

if (def.displayName === 'Average') {
        throw new Error('simulated runtime error!');
      }

With this simulated error, the lens panels with the Average operation will not render, and the smoke test will stall waiting for the dashboard to render. This will fail the test.

Other error cases covered

If a Lens panel is able to render, but renders as an ErrorEmbeddable the smoke test will fail.
If a Lens panel is able to correctly render, but does not have an edit action when the dashboard is in edit mode, the test will fail. This is often a sign of something else going wrong.

Going forward

Short Term

Short term, more of this style of smoke test will be written, covering Visualize and Maps migrations from 7.12.1. This way all combinations of by value, by reference, and enhanced panels will have test coverage going forward. The process for creating these smoke tests will be as follows:

1. Bootstrap and start Kibana 7.12.1 locally
2. Use esArchiver to load in a dataset that can be easily used in functional tests, for instance:
   • node scripts/es_archiver load getting_started/shakespeare --config test/functional/config.js --es-url=http://localhost:9200 --kibana-url=http://localhost:5601
3. Build a representative dashboard with a representative sample of all the 7.12.1 features on that panel type.
4. Export that dashboard.
5. Add the ndjson export to the x-pack/test/functional/apps/dashboard/migration_smoke_tests/exports folder
6. Use the Lens smoke tests as an example of how to import the dashboard and cover its features.

Long Term

In the past, we have had a process where when creating new features, functional tests which cover those features usually create new saved objects during the course of the test, using whatever the latest version happens to be.

This process needs to change for the long term, to promote the creation of additional migration smoke tests like this one for all new features which have some component of state persistence. A sample flow for this kind of process would be:

1. Once a new feature is in a stable state, (before public release), build a dashboard using the feature.
2. Export that dashboard to an .ndjson file.
3. Store that ndjson in the repo. This effectively freezes the state to be exactly as it was when created.
4. Write functional tests which import that saved object, then run smoke tests tests against it.

Ideally, we could get to a state where each new feature that persists state runs exactly the same functional test suite on state which has been migrated from the version in which it was initially created, and on state that is newly created in the current version.

PR Items

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
There may be combinations of Lens state not covered in this smoke test	Low	High	There is fairly good representation of lens states, but this PR does not cover each individual opteration type which lens uses. Instead, this PR focuses on capturing the common ways that errors are expressed, such as an error which fails the migration entirely, an error which stops the panel from rendering, or an error that stops the panel from being edited.

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[ThomThomson]

@elasticmachine merge upstream

[jloleysens]

Great job @ThomThomson ! Thanks for adding this test coverage. I left a couple comments that I'd like to get your thoughts on.

[jloleysens]

I spoke with @tylersmalley about how we might load legacy saved object data into Kibana and he mentioned we should use the kbnArchiver which is a tool that aims to replace esArchiver and other methods for loading of saved objects. Example usage:

kibana/test/api_integration/apis/saved_objects/bulk_update.ts

Line 19 in 3860486

await kibanaServer.importExport.load(

This will also interact with Kibana via the API rather than the UI to load data which should be faster. Let me know what you think!

[ThomThomson]

I will look into this! Anything that speeds up the run is a good thing. I assume that this can support ndjson files?

[jloleysens]

I believe so! The kbn_archiver.js script also allows for extracting those values from the cluster. For example:

node scripts/kbn_archiver.js --config test/functional/config.js save test/functional/fixtures/kbn_archiver/foo.json --type index-pattern,dashboard

[ThomThomson]

In speaking to @tylersmalley, it like it doesn't support ndjson, and I would need to load up 7.12 again and re-export. It's probably worth mentioning this in the description so that people know it's an option going forward, but I don't think it's necessary to update to this format on this PR.

[tylersmalley]

I have an open issue to add support for a pretty format to the saved object export which would allow us to directly support that format: #103021

[jloleysens]

Should this not happen as part of the before setup? I might be missing something though!

[ThomThomson]

It could be a part of the before setup because it is required by all subsequent tests - I moved it into its own test because the test fails if a migration error occurs, and I wanted the failed test name to be more explicit. I will take a look into moving most of this into before but leaving the import success validation as its own test.

[jloleysens]

Thanks for adding these tests @ThomThomson ! I will approve to unblock your progress 👍🏻

[poffdeluxe]

LGTM!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 216b0c7
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #132712 succeeded 2faf214
• 💚 Build #132528 succeeded d80239b
• 💔 Build #132508 failed 49c5ddd5d9c455ab211a07aab6131d13a4b0ee0d
• 💔 Build #132506 failed beda7698e0b9f0be65789b7e7da8b57b65d6cf01
• 💔 Build #132492 failed 1374594

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.