[BWC] OpenSearch Dashboards break version parity with OpenSearch

[tmarkley]

[UPDATE] We probably won't be releasing OpenSearch Dashboards 2.0 with OpenSearch 1.3

Retitled from: OpenSearch Dashboards compatible with older version of OpenSearch

Description

Our plan is to remain compatible with 1.x when we bump to 2.0. In terms of compatibility, this would mean runtime of OpenSearch Dashboards 2.0 to OpenSearch 1.x.

At least ignoreVersionMismatch we can generally see min OpenSearch Dashboards works fine with min OpenSearch. We haven't changed anything with APIs or saved objects. So it should generally work out of the box but we can't assume that this is valid for external plugins. For example, OpenSearch Dashboards Security plugin 2.0 might not be compatible with OpenSearch Security plugin 1.x. If so then, technically then OpenSearch Dashboards distributions won't be compatible.

This work will be defining if this means do we only guarantee support for min OpenSearch Dashboards to version mismatches. And if we wish to change the out of the box experience to support version mismatches. Or if it means distributions of OpenSearch Dashboards (meaning with custom plugins) need to support version mismatches. If so then it will require onboarding plugins to the architectural approach and verifying if 2.0 distributions can connect to OpenSearch 1.x distributions.

Pre-work

• Verify with BWC test script for min OpenSearch Dashboards 2.0.0 works with min OpenSearch 2.0 and 1.x ✅
  • Successful, however I had to remove this conditional check [HERE] and then rebuilt a min. Not positive of the full consequences of removing this check. Then set opensearch.ignoreVersionMismatch: true in the opensearch_dashboards.yml. After that was able to start and it interacts fine.
• Verify with BWC test script for dist OpenSearch Dashboards 2.0.0 works with dist OpenSearch 2.0 and 1.x ✅
  • Results
  • Updating the plugins to 2.0.0.0 build the zips of the plugins and install to the min build. Application works and starts and ran the sanity tests for the results above.
• Determine what full compatibility is as well, for example, do we support removing of deprecated configs?
• Connect with plugin owners

Options (depending results)

• Make the announcement to stakeholders, docs, and plugin owners if OpenSearch Dashboards dist succeeds BWC tests, integ tests, and smoke tests then
• Only support min OpenSearch Dashboards compatibility of 2.0.0 to current and previous versions of OpenSearch but release OpenSearch Dashboards 2.0.0 only with OpenSearch 2.0.0 if OpenSearch Dashboards dist fails BWC tests, integ tests, or smoke tests fail
• Do not support min or dist OpenSearch Dashboards compatibility of 2.0.0 to current and previous versions of OpenSearch if OpenSearch Dashboards dist fails BWC tests, integ tests, or smoke tests fail.

Post-work

If the ability to run BWC tests then we should determine if we want to take releases case by case. Like if we determine 2.0.0 OpenSearch Dashboards is fully compatible but for some reason 2.x isn't will we pick and chose which releases are BWC? This might introduce some confusion compared to just being in lockstep with versions with OpenSearch.

Tasks

• [2.0-COMPAT] Research for Dashboards 2.0 compatibility with OpenSearch 1.x #1101
• Proposal

[kavilla]

Do we consider this in our realm of of compatibility: #533

[kavilla]

Alerting Dashboards

cypress run --env security_enabled=$SECURITY_ENABLED opensearch_dashboards=${BIND_ADDRESS}:${BIND_PORT}

Results:
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  alert_spec.js                            02:13        5        5        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  bucket_level_monitor_spec.js             00:44        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  destination_spec.js                      00:12        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  query_level_monitor_spec.js              00:17        4        4        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        03:27       17       17        -        -        -  

Index Management Dashboards

cypress run --env security_enabled=$SECURITY_ENABLED opensearch_dashboards=${BIND_ADDRESS}:${BIND_PORT}

Results:
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  indices_spec.js                          00:20        3        3        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  managed_indices_spec.js                  01:04        5        5        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  policies_spec.js                         00:19        5        5        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  rollups_spec.js                          00:23        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  transforms_spec.js                       00:19        4        4        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        02:27       21       21        -        -        -  

Workbench

cypress run --env security_enabled=$SECURITY_ENABLED opensearch_dashboards=${BIND_ADDRESS}:${BIND_PORT}

Results:
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  ui.spec.js                               01:23       20       20        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        01:23       20       20        -        -        -  

Dashboards Reports

yarn cypress:run

Results:
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  01-create.spec.ts                        02:04       15       15        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  02-edit.spec.ts                          01:21        3        3        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  03-details.spec.ts                       00:33        6        6        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  04-download.spec.ts                      01:46        5        3        2        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 4 failed (25%)                      05:45       29       27        2        -        -  

Observability Dashboards

yarn cypress:run

Results: 
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  event_analytics.spec.js                  01:27       17       17        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  notebooks.spec.js                        05:32       28       27        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  panels.spec.js                           07:47       26       21        5        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  trace_analytics_dashboard.spec.js        01:03        8        8        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  trace_analytics_services.spec.js         00:31        6        6        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  trace_analytics_traces.spec.js           00:28        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  2 of 6 failed (33%)                      16:51       90       84        6        -        -  

error Command failed with exit code 6.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Gantt charts

yarn cypress:run

Results:
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  ui.spec.js                               02:15        9        9        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        02:15        9        9        -        -        -  

Done in 146.67s.

AD

yarn cy:run-with-security

Results:
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  ad/dashboard/ad_dashboard.spec.ts        00:07        6        6        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  ad/detectorList/detector_list.spec.      00:11       10        9        -        1        - │
  │    ts                                                                                          │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  ad/workflow/create_detector.spec.ts        7ms        1        -        -        1        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:19       17       15        -        2        -  

Done in 31.47s.

[kavilla]

For Observability plugin, the errors related to panels were due to inconsistency. After manually doing those tests I was able to recreate the steps. However, I keep getting an error related to puppeteer or hapi when trying to download a png. But the step before has it succeeding while downloading a PDF. This might be a local setup issue will need a secondary opinion.

[seanneumann]

FYI - opening a doc issue to track compatibility: opensearch-project/documentation-website#366

[kavilla]

Another thing to think about is that if we plan on removing any configurations. For example, the non-inclusive terms (whitelist/blacklist). If OpenSearch Dashboards releases 2.0 with OpenSearch 1.3 but everything is compatible API and doc-wise but we removed the setting would that make the release as a whole for OpenSearch 1.3 not really a minor change?

[kavilla]

Also need to think about documentation and how that looks like. Will docs always be based on the OpenSearch release? So if OpenSearch releases 1.3 with OpenSearch Dashboards it should be indicated that the documentation for OpenSearch Dashboards is for v2.0.0.

[kavilla]

Will update to reflect it as a future idea but won't be achieved for 2.0.0

[kavilla]

Assigning @AMoo-Miki, pinging you to see if this issue is a good issue to re-purpose.