Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MDS] Observability Datasource Plugin migration with MDS support #7143

Merged
merged 19 commits into from
Jul 1, 2024

Conversation

RyanL1997
Copy link
Contributor

@RyanL1997 RyanL1997 commented Jul 1, 2024

Signed-off-by: Ryan Liang jiallian@amazon.com

Description

Observability Datasource Plugin migration with MDS support

Issues Resolved

  • Introduced a centralized panel for data source management plugin for the migration of zero-ETL datasource
  • Introduced a centralized panel for data source creation
  • Introduced MDS support for zero-ETL datasource
  • Change the registration behavior of datasource management plugin

Demo

Testing the changes

Feature flag enabled:

Screen.Recording.2024-07-01.at.11.06.28.AM.mov

Feature flag disabled:

Screen.Recording.2024-07-01.at.3.20.09.PM.mov

Changelog

  • feat: [MDS] Observability Datasource Plugin migration with MDS support

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
  • New functionality includes testing.
  • New functionality has been documented.
  • Update CHANGELOG.md
  • Commits are signed per the DCO using --signoff

Signed-off-by: Ryan Liang <jiallian@amazon.com>
…ource

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: Ryan Liang <jiallian@amazon.com>
Copy link
Contributor

github-actions bot commented Jul 1, 2024

ℹ️ Manual Changeset Creation Reminder

Please ensure manual commit for changeset file 7143.yml under folder changelogs/fragments to complete this PR.

If you want to use the available OpenSearch Changeset Bot App to avoid manual creation of changeset file you can install it in your forked repository following this link.

For more information about formatting of changeset files, please visit OpenSearch Auto Changeset and Release Notes Tool.

Copy link
Contributor

github-actions bot commented Jul 1, 2024

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

Copy link

codecov bot commented Jul 1, 2024

Codecov Report

Attention: Patch coverage is 70.47619% with 93 lines in your changes missing coverage. Please review.

Project coverage is 67.46%. Comparing base (ddf1a41) to head (2c93162).

Files Patch % Lines
...figuration/configure_direct_query_data_sources.tsx 53.94% 30 Missing and 5 partials ⚠️
...ion/manage_direct_query_data_connections_table.tsx 69.81% 12 Missing and 4 partials ⚠️
...guration/direct_query_data_source_auth_details.tsx 61.90% 7 Missing and 1 partial ⚠️
...tion/amazon_s3/configure_amazon_s3_data_source.tsx 63.15% 6 Missing and 1 partial ⚠️
...rc/plugins/data_source_management/public/plugin.ts 53.84% 1 Missing and 5 partials ⚠️
...nnection/direct_query_data_source_delete_modal.tsx 50.00% 3 Missing and 1 partial ⚠️
...irect_query_data_source_configuration/name_row.tsx 86.36% 2 Missing and 1 partial ⚠️
...on/prometheus/configure_prometheus_data_source.tsx 72.72% 3 Missing ⚠️
...ry_data_source_configuration/query_permissions.tsx 78.57% 1 Missing and 2 partials ⚠️
...uration/amazon_s3/review_amazon_s3_data_source.tsx 66.66% 0 Missing and 2 partials ⚠️
... and 4 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7143      +/-   ##
==========================================
- Coverage   67.48%   67.46%   -0.02%     
==========================================
  Files        3451     3468      +17     
  Lines       68066    68363     +297     
  Branches    11069    11110      +41     
==========================================
+ Hits        45932    46119     +187     
- Misses      19455    19540      +85     
- Partials     2679     2704      +25     
Flag Coverage Δ
Linux_1 33.04% <33.33%> (+<0.01%) ⬆️
Linux_2 55.18% <ø> (ø)
Linux_3 45.35% <70.83%> (+0.21%) ⬆️
Linux_4 34.77% <41.66%> (+0.01%) ⬆️
Windows_1 ?
Windows_2 55.14% <ø> (ø)
Windows_3 ?
Windows_4 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

github-actions bot commented Jul 1, 2024

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

1 similar comment
Copy link
Contributor

github-actions bot commented Jul 1, 2024

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

Copy link
Contributor

github-actions bot commented Jul 1, 2024

❌ Entry Too Long

Entry is 146 characters long, which is 46 characters longer than the maximum allowed length of 100 characters. Please revise your entry to be within the maximum length.

Copy link
Member

@zhongnansu zhongnansu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RyanL1997 Why in the "disable feature flag" video, there's still MDS create data source available? I thought the purpose is the hide/disable "create data source", when data_source.enabled: false, or this will be the next step?

@RyanL1997
Copy link
Contributor Author

RyanL1997 commented Jul 1, 2024

Hi @zhongnansu, good catch. I uploaded the wrong video... and I just corrected that in PR description.

@mengweieric mengweieric merged commit 9ac3184 into opensearch-project:main Jul 1, 2024
68 of 79 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7143-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9ac318456ce93714223b801953fa5973f855bf60
# Push it to GitHub
git push --set-upstream origin backport/backport-7143-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7143-to-2.x.

RyanL1997 added a commit to RyanL1997/OpenSearch-Dashboards that referenced this pull request Jul 22, 2024
…nsearch-project#7143)

* Observability Datasource Plugin migration with MDS support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Switch to use doc services for doc links of configuration of s3 datasource

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for home panel

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the snapshot for mds

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for data source creation panel

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for dq data connection table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for dq configuration

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add readonly for s3glue doc link

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for s3 prometheus creation and review

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix mount feature flag behavior and tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Shorten file names

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Shorten file names again

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Experiment 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the visbuilder failure 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the setter of datasource setup in vis_type_timeseries

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Synced branch

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the readonly in doc link and add change log

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the create opensearch datasource's cancel button redirection

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more test cases for breadscrumb changes and mount behavior based on the plugin registration

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
BionIT pushed a commit that referenced this pull request Jul 22, 2024
…S support for Data Connection Table (#7371)

* [MDS] Observability Datasource Plugin migration with MDS support (#7143)

* Observability Datasource Plugin migration with MDS support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Switch to use doc services for doc links of configuration of s3 datasource

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for home panel

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the snapshot for mds

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for data source creation panel

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for dq data connection table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for dq configuration

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add readonly for s3glue doc link

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for s3 prometheus creation and review

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix mount feature flag behavior and tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Shorten file names

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Shorten file names again

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Experiment 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the visbuilder failure 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the setter of datasource setup in vis_type_timeseries

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Synced branch

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the readonly in doc link and add change log

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the create opensearch datasource's cancel button redirection

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more test cases for breadscrumb changes and mount behavior based on the plugin registration

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* [MDS] Fix the dsm plugin setup when mds feature flag is disabled (#7163)

* Fix the dsm plugin setup when mds feature flag is disabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Changeset file for PR #7163 created/updated

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* Changeset file for PR #7371 created/updated

* Fix snapshots

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 23, 2024
…S support for Data Connection Table (#7371)

* [MDS] Observability Datasource Plugin migration with MDS support (#7143)

* Observability Datasource Plugin migration with MDS support

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Switch to use doc services for doc links of configuration of s3 datasource

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for home panel

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the snapshot for mds

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for data source creation panel

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for dq data connection table

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for dq configuration

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add readonly for s3glue doc link

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add tests for s3 prometheus creation and review

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix mount feature flag behavior and tests

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Shorten file names

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Shorten file names again

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Experiment 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the visbuilder failure 1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the setter of datasource setup in vis_type_timeseries

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Synced branch

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the readonly in doc link and add change log

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Fix the create opensearch datasource's cancel button redirection

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Add more test cases for breadscrumb changes and mount behavior based on the plugin registration

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* [MDS] Fix the dsm plugin setup when mds feature flag is disabled (#7163)

* Fix the dsm plugin setup when mds feature flag is disabled

Signed-off-by: Ryan Liang <jiallian@amazon.com>

* Changeset file for PR #7163 created/updated

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* Changeset file for PR #7371 created/updated

* Fix snapshots

Signed-off-by: Ryan Liang <jiallian@amazon.com>

---------

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
(cherry picked from commit 704f2fa)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ruanyl pushed a commit that referenced this pull request Jul 23, 2024
…S support for Data Connection Table (#7371) (#7384)

* [MDS] Observability Datasource Plugin migration with MDS support (#7143)

* Observability Datasource Plugin migration with MDS support



* Switch to use doc services for doc links of configuration of s3 datasource



* Add tests for home panel



* Fix the snapshot for mds



* Add tests for data source creation panel



* Add tests for dq data connection table



* Add tests for dq configuration



* Add readonly for s3glue doc link



* Add tests for s3 prometheus creation and review



* Fix mount feature flag behavior and tests



* Shorten file names



* Shorten file names again



* Experiment 1



* Fix the visbuilder failure 1



* Fix the setter of datasource setup in vis_type_timeseries



* Synced branch



* Fix the readonly in doc link and add change log



* Fix the create opensearch datasource's cancel button redirection



* Add more test cases for breadscrumb changes and mount behavior based on the plugin registration



---------



* [MDS] Fix the dsm plugin setup when mds feature flag is disabled (#7163)

* Fix the dsm plugin setup when mds feature flag is disabled



* Changeset file for PR #7163 created/updated

---------




* Changeset file for PR #7371 created/updated

* Fix snapshots



---------



(cherry picked from commit 704f2fa)

Signed-off-by: Ryan Liang <jiallian@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
@BionIT
Copy link
Collaborator

BionIT commented Jul 23, 2024

@RyanL1997 failed backport, did we do manual backport to release branch

@AMoo-Miki
Copy link
Collaborator

Looks like 2.x was manually backported and that was backported to 2.16 successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants