[Tests] Add Github workflow for Test Orchestrator in FT Repo to run cypress tests within Dashboards repo #5725
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| name: Orchestrator cypress workflow | ||
| run-name: dashboards_cypress_workflow ${{ inputs.UNIQUE_ID != '' && inputs.UNIQUE_ID || '' }} # Unique id number appended to the workflow run-name to reference the run within the orchestrator. | ||
| # Trigger on dispatch event sent from FT repo orchestrator | ||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| test_repo: | ||
| description: 'Cypress test repo' | ||
| default: '' | ||
| required: false | ||
| type: string | ||
| test_branch: | ||
| description: 'Cypress test branch (default: source branch)' | ||
| required: false | ||
| type: string | ||
| specs: | ||
| description: 'Test group to run' | ||
| required: false | ||
| type: string | ||
| build_id: | ||
| description: 'Build Id' | ||
| required: false | ||
| type: string | ||
| OS_URL: | ||
| description: 'OpenSearch release artifact' | ||
| required: false | ||
| type: string | ||
| OSD_URL: | ||
| description: 'OpenSearch Dashboards release artifact' | ||
| required: false | ||
| type: string | ||
| UNIQUE_ID: | ||
| description: 'Unique Id for the workflow execution' | ||
| required: true | ||
| type: string | ||
| SECURITY_ENABLED: | ||
| required: false | ||
| type: string | ||
|
|
||
| env: | ||
| TEST_REPO: ${{ inputs.test_repo != '' && inputs.test_repo || github.repository }} | ||
| TEST_BRANCH: "${{ inputs.test_branch != '' && inputs.test_branch || github.base_ref }}" | ||
| OSD_PATH: 'osd' | ||
| CYPRESS_BROWSER: 'chromium' | ||
| JOB_ID: ${{ inputs.UNIQUE_ID}} | ||
| OPENSEARCH: ${{ inputs.OS_URL != '' && inputs.OS_URL || 'https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/$VERSION/latest/linux/x64/tar/dist/opensearch/opensearch-$VERSION-linux-x64.tar.gz' }} | ||
|
There was a problem hiding this comment. would we be able to do a matrix and pass in like linux, windows and x64, arm |
||
| DASHBOARDS: ${{ inputs.OSD_URL != '' && inputs.OSD_URL || 'https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/$VERSION/latest/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-$VERSION-linux-x64.tar.gz' }} | ||
| OPENSEARCH_DIR: 'cypress/opensearch' | ||
| DASHBOARDS_DIR: 'cypress/opensearch-dashboards' | ||
| SECURITY_ENABLED: ${{ inputs.SECURITY_ENABLED != '' && inputs.SECURITY_ENABLED || 'false' }} | ||
| SPEC: 'cypress/integration/core_opensearch_dashboards/*' | ||
|
|
||
| jobs: | ||
| cypress-tests: | ||
| runs-on: ubuntu-latest | ||
| container: | ||
| image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2 | ||
|
There was a problem hiding this comment. do we want to consider using this: https://github.com/opensearch-project/dashboards-assistant/blob/main/.github/workflows/unit_test_workflow.yml#L17 which has some pre-setup. One con would be that if we bump a dependency (like Node) the tests will fail since it won't find the required Node version if it isnt already installed |
||
| options: --user 1001 | ||
| env: | ||
| # prevents extra Cypress installation progress messages | ||
| CI: 1 | ||
| # avoid warnings like "tput: No value for $TERM and no -T specified" | ||
| TERM: xterm | ||
| name: Run cypress tests ${{ inputs.UNIQUE_ID}} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v2 | ||
| with: | ||
| path: ./${{ env.OSD_PATH }} | ||
| repository: ${{ env.TEST_REPO }} | ||
| ref: '${{ env.TEST_BRANCH }}' | ||
|
|
||
| - name: Setup Node | ||
| uses: actions/setup-node@v3 | ||
| with: | ||
| node-version-file: './${{ env.OSD_PATH }}/.nvmrc' | ||
| registry-url: 'https://registry.npmjs.org' | ||
|
|
||
| - name: Setup Yarn | ||
| run: | | ||
| npm uninstall -g yarn | ||
| npm i -g yarn@1.22.10 | ||
| yarn config set network-timeout 1000000 -g | ||
|
|
||
| - name: Get Cypress version | ||
| id: cypress_version | ||
| run: | | ||
| echo "name=cypress_version::$(cat ./${{ env.OSD_PATH }}/package.json | jq '.devDependencies.cypress' | tr -d '"')" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Cache Cypress | ||
|
There was a problem hiding this comment. totally unrelated: but does this cache help the re-run of stuff? |
||
| id: cache-cypress | ||
| uses: actions/cache@v1 | ||
| with: | ||
| path: ~/.cache/Cypress | ||
| key: cypress-cache-v2-${{ runner.os }}-${{ hashFiles('**/package.json') }} | ||
| env: | ||
| CYPRESS_INSTALL_BINARY: ${{ steps.cypress_version.outputs.cypress_version }} | ||
| - run: npx cypress cache list | ||
| - run: npx cypress cache path | ||
|
|
||
| - name: Get package version (Linux) | ||
| run: | | ||
| cd ${{ env.OSD_PATH }} | ||
| echo "VERSION=$(yarn --silent pkg-version)" >> $GITHUB_ENV | ||
|
|
||
| - name: Run bootstrap | ||
| run: | | ||
| cd ${{ env.OSD_PATH }} | ||
| yarn osd bootstrap | ||
|
|
||
| - name: Download and extract Opensearch artifacts | ||
|
|
||
| run: | | ||
| CWD=$(pwd) | ||
| mkdir -p $CWD/${{ env.OPENSEARCH_DIR }} | ||
| source ${{ env.OSD_PATH }}/scripts/common/utils.sh | ||
| open_artifact $CWD/${{ env.OPENSEARCH_DIR }} ${{ env.OPENSEARCH }} | ||
|
|
||
| - name: Download and extract Opensearch Dashboards artifacts | ||
|
|
||
| run: | | ||
| CWD=$(pwd) | ||
| mkdir -p $CWD/${{ env.DASHBOARDS_DIR }} | ||
| source ${{ env.OSD_PATH }}/scripts/common/utils.sh | ||
| open_artifact $CWD/${{ env.DASHBOARDS_DIR }} ${{ env.DASHBOARDS }} | ||
|
|
||
| - name: Run Cypress tests | ||
| run: | | ||
| source ${{ env.OSD_PATH }}/scripts/cypress_tests.sh | ||
| run_dashboards_cypress_tests | ||
|
|
||
| # Screenshots are only captured on failures | ||
| - uses: actions/upload-artifact@v3 | ||
| if: failure() | ||
| with: | ||
| name: osd-cypress-screenshots | ||
|
There was a problem hiding this comment. same as the above comment: indicate that these tests were executed against a release artifact so |
||
| path: ${{ env.OSD_PATH }}/cypress/screenshots | ||
| retention-days: 1 | ||
|
|
||
| - uses: actions/upload-artifact@v3 | ||
| if: always() | ||
| with: | ||
| name: osd-cypress-videos | ||
| path: ${{ env.OSD_PATH }}/cypress/videos | ||
| retention-days: 1 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| const { defineConfig } = require('cypress'); | ||
|
|
||
| module.exports = defineConfig({ | ||
|
There was a problem hiding this comment. IMO, I think if we are adding this already we should create a Sort of how like selenium does it where it defines the config at the functional folder level and adds subdirectories below that. |
||
| defaultCommandTimeout: 60000, | ||
| requestTimeout: 60000, | ||
| responseTimeout: 60000, | ||
| baseUrl: 'http://localhost:5601', | ||
| viewportWidth: 2000, | ||
| viewportHeight: 1320, | ||
| env: { | ||
| openSearchUrl: 'http://localhost:9200', | ||
| SECURITY_ENABLED: false, | ||
| AGGREGATION_VIEW: false, | ||
| username: 'admin', | ||
| password: 'myStrongPassword123!', | ||
| ENDPOINT_WITH_PROXY: false, | ||
| MANAGED_SERVICE_ENDPOINT: false, | ||
| VISBUILDER_ENABLED: true, | ||
| DATASOURCE_MANAGEMENT_ENABLED: false, | ||
| ML_COMMONS_DASHBOARDS_ENABLED: true, | ||
| WAIT_FOR_LOADER_BUFFER_MS: 0, | ||
|
Comment on lines
+17
to
+26
There was a problem hiding this comment. It seems only There was a problem hiding this comment. Yes but as the plan is to right away start writing new tests in this repo, I've added/matched mostly what is in FTRepo. There was a problem hiding this comment. do we have a link we can provide with the plan and attach it here? |
||
| }, | ||
| }); | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| /* | ||
|
There was a problem hiding this comment. NIT: I think we don't need the There was a problem hiding this comment. One thing I think will be very important that will be easier to do starting off is to build in the concept of CiGroups like the selenium tests have. For the ftrepo I did something like this: https://github.com/opensearch-project/opensearch-dashboards-functional-test/blob/main/package.json#L28. If we define a folder structure that makes that a little bit easier to do that might be helpful but if this in code solution that would be great too. Cypress tests support parallelization but I don't think we are built out to run the tests like that. |
||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { | ||
| CommonUI, | ||
| MiscUtils, | ||
| } from '@opensearch-dashboards-test/opensearch-dashboards-test-library'; | ||
|
|
||
| /** | ||
| * dashboard_sample_data test suite description: | ||
| * 1) Visit the home page of opensearchdashboard, check key UI elements display | ||
| * 2) add sample data of eCommerce, flights, web logs from tutorial page | ||
| * 3) check each sample data dashboard key UI elements display | ||
| */ | ||
| export function dashboardSanityTests() { | ||
| const commonUI = new CommonUI(cy); | ||
| const miscUtils = new MiscUtils(cy); | ||
| const baseURL = new URL(Cypress.config().baseUrl); | ||
| // remove trailing slash | ||
| const path = baseURL.pathname.replace(/\/$/, ''); | ||
|
|
||
| describe('dashboard sample data validation', () => { | ||
| before(() => {}); | ||
|
|
||
| after(() => {}); | ||
|
|
||
| describe('checking home page', () => { | ||
| before(() => { | ||
| // Go to the home page | ||
| miscUtils.visitPage('app/home#'); | ||
| cy.window().then((win) => win.localStorage.setItem('home:welcome:show', false)); | ||
| cy.reload(true); | ||
| }); | ||
|
|
||
| after(() => { | ||
| cy.window().then((win) => win.localStorage.removeItem('home:welcome:show')); | ||
| }); | ||
|
|
||
| it('checking opensearch_dashboards_overview display', () => { | ||
| // Check that opensearch_dashboards_overview is visable | ||
| commonUI.checkElementExists(`a[href="${path}/app/opensearch_dashboards_overview"]`, 1); | ||
| }); | ||
|
|
||
| it('checking tutorial_directory display', () => { | ||
| // Check that tutorial_directory is visable | ||
| commonUI.checkElementExists(`a[href="${path}/app/home#/tutorial_directory"]`, 2); | ||
| }); | ||
|
|
||
| it('checking management display', () => { | ||
| // Check that management is visable | ||
| commonUI.checkElementExists(`a[href="${path}/app/management"]`, 1); | ||
| }); | ||
|
|
||
| it('checking dev_tools display', () => { | ||
| // Check that dev_tools is visable | ||
| commonUI.checkElementExists(`a[href="${path}/app/dev_tools#/console"]`, 2); | ||
| }); | ||
|
|
||
| it('settings display', () => { | ||
| // Check that settings is visable | ||
| commonUI.checkElementExists( | ||
| `a[href="${path}/app/management/opensearch-dashboards/settings#defaultRoute"]`, | ||
| 1 | ||
| ); | ||
| }); | ||
|
|
||
| it('checking feature_directory display', () => { | ||
| // Check that feature_directory is visable | ||
| commonUI.checkElementExists(`a[href="${path}/app/home#/feature_directory"]`, 1); | ||
| }); | ||
|
|
||
| it('checking navigation display', () => { | ||
| // Check that navigation is visable | ||
| commonUI.checkElementExists('button[data-test-subj="toggleNavButton"]', 1); | ||
| }); | ||
|
|
||
| it('checking Help menu display', () => { | ||
| // Check that Help menu is visable | ||
| commonUI.checkElementExists('button[aria-label="Help menu"]', 1); | ||
| }); | ||
| }); | ||
|
|
||
| describe('checking Dev Tools', () => { | ||
| before(() => { | ||
| // Go to the Dev Tools page | ||
| miscUtils.visitPage('app/dev_tools#/console'); | ||
| }); | ||
|
|
||
| after(() => {}); | ||
|
|
||
| it('checking welcome panel display', () => { | ||
| commonUI.checkElementExists('div[data-test-subj="welcomePanel"]', 1); | ||
| }); | ||
|
|
||
| it('checking dismiss button display', () => { | ||
| commonUI.checkElementExists('button[data-test-subj="help-close-button"]', 1); | ||
| }); | ||
|
|
||
| it('checking console input area display', () => { | ||
| commonUI.checkElementExists('div[data-test-subj="request-editor"]', 1); | ||
| }); | ||
|
|
||
| it('checking console output area display', () => { | ||
| commonUI.checkElementExists('div[data-test-subj="response-editor"]', 1); | ||
| }); | ||
| }); | ||
|
|
||
| describe('adding sample data', () => { | ||
| before(() => { | ||
| miscUtils.addSampleData(); | ||
| }); | ||
|
|
||
| after(() => { | ||
| miscUtils.removeSampleData(); | ||
| }); | ||
|
|
||
| it('checking ecommerce dashboards displayed', () => { | ||
| miscUtils.viewData('ecommerce'); | ||
| commonUI.checkElementContainsValue( | ||
| 'span[title="[eCommerce] Revenue Dashboard"]', | ||
| 1, | ||
| '\\[eCommerce\\] Revenue Dashboard' | ||
| ); | ||
| commonUI.checkElementContainsValue( | ||
| 'div[data-test-subj="markdownBody"] > h3', | ||
| 1, | ||
| 'Sample eCommerce Data' | ||
| ); | ||
| }); | ||
|
|
||
| it('checking flights dashboards displayed', () => { | ||
| miscUtils.viewData('flights'); | ||
| commonUI.checkElementContainsValue( | ||
| 'span[title="[Flights] Global Flight Dashboard"]', | ||
| 1, | ||
| '\\[Flights\\] Global Flight Dashboard' | ||
| ); | ||
| commonUI.checkElementContainsValue( | ||
| 'div[data-test-subj="markdownBody"] > h3', | ||
| 1, | ||
| 'Sample Flight data' | ||
| ); | ||
| }); | ||
|
|
||
| it('checking web logs dashboards displayed', () => { | ||
| miscUtils.viewData('logs'); | ||
| commonUI.checkElementContainsValue( | ||
| 'span[title="[Logs] Web Traffic"]', | ||
| 1, | ||
| '\\[Logs\\] Web Traffic' | ||
| ); | ||
| commonUI.checkElementContainsValue( | ||
| 'div[data-test-subj="markdownBody"] > h3', | ||
| 1, | ||
| 'Sample Logs Data' | ||
| ); | ||
| }); | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| dashboardSanityTests(); | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -77,7 +77,10 @@ | |
| "docs:acceptApiChanges": "scripts/use_node --max-old-space-size=6144 scripts/check_published_api_changes.js --accept", | ||
| "osd:bootstrap": "scripts/use_node scripts/build_ts_refs && scripts/use_node scripts/register_git_hook", | ||
| "spec_to_console": "scripts/use_node scripts/spec_to_console", | ||
| "pkg-version": "scripts/use_node -e \"console.log(require('./package.json').version)\"", | ||
| "cypress:run-without-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false", | ||
| "cypress:run-with-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=true,openSearchUrl=https://localhost:9200,WAIT_FOR_LOADER_BUFFER_MS=500" | ||
|
|
||
| }, | ||
| "repository": { | ||
| "type": "git", | ||
|
|
@@ -226,7 +229,9 @@ | |
| "type-detect": "^4.0.8", | ||
| "uuid": "3.3.2", | ||
| "whatwg-fetch": "^3.0.0", | ||
| "yauzl": "^2.10.0", | ||
| "@opensearch-dashboards-test/opensearch-dashboards-test-library": "https://github.com/opensearch-project/opensearch-dashboards-test-library/archive/refs/tags/1.0.6.tar.gz" | ||
|
There was a problem hiding this comment. i'm fine for this for now. but the refs was a painful thing anytime we need to make a change and if we plan on adopting this more i'd imagine it would be helpful to build out the library within this repo and other folks can rely on it being available in OSD instead of an external dependency that needs to be re-installed. For example, if we just take what's in that package and dump it here: https://github.com/opensearch-project/OpenSearch-Dashboards/tree/main/packages/osd-test in a clean folder structure then we can fix it as we go. or add stuff that can be reusable. |
||
|
|
||
| }, | ||
| "devDependencies": { | ||
| "@babel/core": "^7.22.9", | ||
|
|
@@ -355,6 +360,7 @@ | |
| "chromedriver": "^119.0.1", | ||
| "classnames": "2.3.1", | ||
| "compare-versions": "3.5.1", | ||
| "cypress": "9.5.4", | ||
SuZhou-Joe marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "d3": "3.5.17", | ||
| "d3-cloud": "1.2.5", | ||
| "dedent": "^0.7.0", | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,9 +5,9 @@ | |
|
|
||
| set -e | ||
|
|
||
| . scripts/common/utils.sh | ||
|
There was a problem hiding this comment. could we create an issue into adding these to: https://github.com/opensearch-project/OpenSearch-Dashboards/tree/main/packages/osd-test |
||
| . scripts/common/opensearch_service.sh | ||
| . scripts/common/opensearch_dashboards_service.sh | ||
| . scripts/bwc/generate_test_data.sh | ||
|
|
||
| # For every release, add sample data and new version below: | ||
|
|
||
File renamed without changes.
File renamed without changes.
File renamed without changes.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is a better name for this file perhaps
release_cypress_workflowsince it will execute the tests within this repo against a release artifact (something that was built with --release). Because it might be confusing to see on execution that it won't pull the latest source code to run it. If the latest release artifact doesn't contain my code then my tests could still fail even though they were updated to address my changes in a PR.