Skip to content

Commit

Permalink
[CI][BWC] add BWC tests to github actions
Browse files Browse the repository at this point in the history
Add backwards compatibility tests to github workflow for PRs.
Also, fix for checking status of OpenSearch Dashboards since
api/status/ is not protected.

Also, enable ability to create test data since migration from
1.0.0 to 3.0.0 is not supported for OpenSearch currently.

Issue:
#1465

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>
  • Loading branch information
kavilla committed May 29, 2022
1 parent 665424e commit b0e4018
Show file tree
Hide file tree
Showing 13 changed files with 307 additions and 21 deletions.
66 changes: 64 additions & 2 deletions .github/workflows/build_and_test_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,68 @@ jobs:
- uses: actions/upload-artifact@v3
if: success()
with:
name: ${{ matrix.name }}
name: ${{ matrix.suffix }}-${{ env.VERSION }}
path: ./artifacts/target/${{ env.ARTIFACT_BUILD_NAME }}
retention-days: 1
retention-days: 1

bwc-tests:
needs: [ build-min-artifact-tests ]
runs-on: ubuntu-latest
name: Run backwards compatibility tests
defaults:
run:
working-directory: ./artifacts
strategy:
matrix:
version: [ osd-2.0.0 ]
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
path: ./artifacts

- run: echo Running backwards compatibility tests for version ${{ matrix.version }}
- run: echo [NOTE] These tests will be ran using Linux x64 release builds without security

- name: Setup Node
uses: actions/setup-node@v2
with:
node-version-file: "./artifacts/.nvmrc"
registry-url: 'https://registry.npmjs.org'

- name: Setup Yarn
run: |
npm uninstall -g yarn
npm i -g yarn@1.22.10
- name: Get package version
run: |
echo "VERSION=$(yarn --silent pkg-version)" >> $GITHUB_ENV
- name: Set OpenSearch URL
run: |
echo "OPENSEARCH_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/${{ env.VERSION }}/latest/linux/x64/tar/dist/opensearch/opensearch-${{ env.VERSION }}-linux-x64.tar.gz" >> $GITHUB_ENV
- name: Verify if OpenSearch is available for version
id: verify-opensearch-exists
run: |
if curl -I -L ${{ env.OPENSEARCH_URL }}; then
echo "::set-output name=version-exists::true"
fi
- name: Skipping tests
if: steps.verify-opensearch-exists.outputs.version-exists != 'true'
run: echo Tests were skipped because an OpenSearch release build does not exist for this version yet!

- name: Download OpenSearch Dashboards
uses: actions/download-artifact@v3
id: download
with:
name: linux-x64-${{ env.VERSION }}
path: ./artifacts/
if: steps.verify-opensearch-exists.outputs.version-exists == 'true'

- name: Run tests
if: steps.verify-opensearch-exists.outputs.version-exists == 'true'
run: |
./bwctest.sh -s false -o ${{ env.OPENSEARCH_URL }} -d ${{ steps.download.outputs.download-path }}/opensearch-dashboards-${{ env.VERSION }}-linux-x64.tar.gz
6 changes: 6 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ To run specific versions' backwards compatibility tests, pass the versions to th

`yarn test:bwc -o [test path to opensearch.tar.gz] -d [test path to opensearch-dashboards.tar.gz] -v "[test versions]"`

To generate test data that we be utilized for future version backwards compatibility tests:

`yarn test:bwc -o [test path to opensearch.tar.gz] -d [test path to opensearch-dashboards.tar.gz] -g true`

This will create an archive of the data based on the OpenSearch Dashboards version you have provided.

### Additional checks
Make sure you run lint checker before submitting a pull request. To run lint checker:
`node scripts/precommit_hook.js --fix`
Expand Down
5 changes: 3 additions & 2 deletions bwctest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

set -e

DEFAULT_VERSIONS="osd-1.1.0,odfe-1.13.2,odfe-1.0.2"
DEFAULT_VERSIONS="osd-2.0.0,"

function usage() {
echo ""
Expand Down Expand Up @@ -75,6 +75,7 @@ done
[ -z "$SECURITY_ENABLED" ] && SECURITY_ENABLED="false"
[ -z "$CREDENTIAL" ] && CREDENTIAL="admin:admin"
[ -z "$CI" ] && CI=1
[ -z "$BWC_VERSIONS" ] && BWC_VERSIONS=$DEFAULT_VERSIONS

# If no OpenSearch build was passed then this constructs the version
if [ -z "$OPENSEARCH" ]; then
Expand All @@ -96,4 +97,4 @@ if [ -z "$OPENSEARCH" ]; then
OPENSEARCH="https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/$VERSION/latest/$PLATFORM/$ARCH/tar/dist/opensearch/opensearch-$VERSION-$PLATFORM-$ARCH.tar.gz"
fi

source scripts/bwctest_osd.sh -b $BIND_ADDRESS -p $BIND_PORT -s $SECURITY_ENABLED -c $CREDENTIAL -o $OPENSEARCH -d $DASHBOARDS -v $DEFAULT_VERSIONS
source scripts/bwctest_osd.sh -b $BIND_ADDRESS -p $BIND_PORT -s $SECURITY_ENABLED -c $CREDENTIAL -o $OPENSEARCH -d $DASHBOARDS -v $BWC_VERSIONS
8 changes: 6 additions & 2 deletions cypress/integration/with-security/check_loaded_data.js
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ describe('check previously loaded data', () => {
.get('[data-test-subj="dashboardListingTitleLink-[Flights]-Global-Flight-Dashboard"]')
.click();
commonUI.removeAllFilters();
commonUI.setDateRange('Dec 1, 2021 @ 00:00:00.000', 'Nov 1, 2021 @ 00:00:00.000');
commonUI.setDateRange(
`Dec 31, ${new Date().getFullYear()} @ 00:00:00.000', 'Nov 1, 2016 @ 00:00:00.000`
);
});

it('Global Flight Dashboard is loaded and funtions correctly', () => {
Expand Down Expand Up @@ -77,7 +79,9 @@ describe('check previously loaded data', () => {
.get('[data-test-subj="dashboardListingTitleLink-[eCommerce]-Revenue-Dashboard"]')
.click();
commonUI.removeAllFilters();
commonUI.setDateRange('Nov 1, 2021 @ 00:00:00.000', 'Nov 1, 2016 @ 00:00:00.000');
commonUI.setDateRange(
`Dec 31, ${new Date().getFullYear()} @ 00:00:00.000', 'Nov 1, 2016 @ 00:00:00.000`
);
});

it('eCommerce Revenue Dashboard is loaded and functions correctly', () => {
Expand Down
86 changes: 86 additions & 0 deletions cypress/integration/with-security/helpers/generate_data.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

import {
MiscUtils,
LoginPage,
} from '@opensearch-dashboards-test/opensearch-dashboards-test-library';

const miscUtils = new MiscUtils(cy);
const loginPage = new LoginPage(cy);

describe('Generating BWC test data with security', () => {
beforeEach(() => {
miscUtils.visitPage('app/management/opensearch-dashboards/settings');
loginPage.enterUserName('admin');
loginPage.enterPassword('admin');
loginPage.submit();
});

afterEach(() => {
cy.clearCookies();
});

it('tenant-switch-modal page should show and be clicked', () => {
cy.get('[data-test-subj="tenant-switch-modal"]');
cy.get('[data-test-subj="confirm"]').click();
});

it('adds advanced settings', () => {
miscUtils.visitPage('app/management/opensearch-dashboards/settings');
cy.get('[data-test-subj="advancedSetting-editField-theme:darkMode"]').click();
cy.get('[data-test-subj="advancedSetting-editField-timeline:default_columns"]').type(
'{selectAll}4'
);
cy.get('[data-test-subj="advancedSetting-editField-timeline:max_buckets"]').type(
'{selectAll}4'
);
cy.get('[data-test-subj="advancedSetting-editField-defaultRoute"]')
.clear()
.type('/app/opensearch_dashboards_overview#/');
cy.get('[data-test-subj="advancedSetting-saveButton"]').click({ force: true });
cy.reload();
});

it('adds sample data', () => {
miscUtils.addSampleData();
});

it('adds filters and queries', () => {
miscUtils.visitPage('app/dashboards#');
cy.get('[data-test-subj="dashboardListingTitleLink-[Logs]-Web-Traffic"]').click();
cy.get('[data-test-subj="queryInput"]').clear().type('resp=200');
cy.get('[data-test-subj="addFilter"]').click();
cy.get('[data-test-subj="filterFieldSuggestionList"]')
.find('[data-test-subj="comboBoxInput"]')
.type('machine.os{downArrow}{enter}');
cy.get('[data-test-subj="filterOperatorList"]')
.find('[data-test-subj="comboBoxInput"]')
.type('{downArrow}{enter}');
cy.get('[data-test-subj="filterParams"]')
.find('input.euiFieldText[placeholder="Enter a value"]')
.type('osx');
cy.get('[data-test-subj="createCustomLabel"]').click();
cy.get('[class="globalFilterItem__editorForm"]').find('input').last().type('osx filter');
cy.get('[data-test-subj="saveFilter"]').click();
cy.get('[data-test-subj="saved-query-management-popover-button"]').click();
cy.get('[data-test-subj="saved-query-management-popover"]')
.find('[data-test-subj="saved-query-management-save-button"]')
.click();
cy.get('[data-test-subj="saveQueryFormTitle"]').type('test-query');
cy.get('[data-test-subj="savedQueryFormSaveButton"]').click();
});

it('adds Timeline visualization', () => {
miscUtils.visitPage('app/visualize#');
cy.get('[data-test-subj="visualizationLandingPage"]')
.find('[data-test-subj="newItemButton"]')
.click();
cy.get('[data-test-subj="visType-timelion"]').click();
cy.get('[data-test-subj="visualizeSaveButton"]').click();
cy.get('[data-test-subj="savedObjectTitle"]').type('test-timeline');
cy.get('[data-test-subj="confirmSaveSavedObjectButton"]').click();
});
});
8 changes: 6 additions & 2 deletions cypress/integration/without-security/check_loaded_data.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@ describe('check previously loaded data', () => {
.get('[data-test-subj="dashboardListingTitleLink-[Flights]-Global-Flight-Dashboard"]')
.click();
commonUI.removeAllFilters();
commonUI.setDateRange('Dec 1, 2021 @ 00:00:00.000', 'Nov 1, 2021 @ 00:00:00.000');
commonUI.setDateRange(
`Dec 31, ${new Date().getFullYear()} @ 00:00:00.000', 'Nov 1, 2016 @ 00:00:00.000`
);
});

it('Global Flight Dashboard is loaded when clicked', () => {
Expand Down Expand Up @@ -71,7 +73,9 @@ describe('check previously loaded data', () => {
.get('[data-test-subj="dashboardListingTitleLink-[eCommerce]-Revenue-Dashboard"]')
.click();
commonUI.removeAllFilters();
commonUI.setDateRange('Nov 1, 2021 @ 00:00:00.000', 'Nov 1, 2016 @ 00:00:00.000');
commonUI.setDateRange(
`Dec 31, ${new Date().getFullYear()} @ 00:00:00.000', 'Nov 1, 2016 @ 00:00:00.000`
);
});

it('eCommerce Revenue Dashboard is loaded when clicked', () => {
Expand Down
69 changes: 69 additions & 0 deletions cypress/integration/without-security/helpers/generate_data.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

import { MiscUtils } from '@opensearch-dashboards-test/opensearch-dashboards-test-library';

const miscUtils = new MiscUtils(cy);

describe('Generating BWC test data without security', () => {
before(() => {
miscUtils.visitPage('app');
});
it('adds advanced settings', () => {
miscUtils.visitPage('app/management/opensearch-dashboards/settings');
cy.get('[data-test-subj="advancedSetting-editField-theme:darkMode"]').click();
cy.get('[data-test-subj="advancedSetting-editField-timeline:default_columns"]').type(
'{selectAll}4'
);
cy.get('[data-test-subj="advancedSetting-editField-timeline:max_buckets"]').type(
'{selectAll}4'
);
cy.get('[data-test-subj="advancedSetting-editField-defaultRoute"]')
.clear()
.type('/app/opensearch_dashboards_overview#/');
cy.get('[data-test-subj="advancedSetting-saveButton"]').click({ force: true });
cy.reload();
});

it('adds sample data', () => {
miscUtils.addSampleData();
});

it('adds filters and queries', () => {
miscUtils.visitPage('app/dashboards#');
cy.get('[data-test-subj="dashboardListingTitleLink-[Logs]-Web-Traffic"]').click();
cy.get('[data-test-subj="queryInput"]').clear().type('resp=200');
cy.get('[data-test-subj="addFilter"]').click();
cy.get('[data-test-subj="filterFieldSuggestionList"]')
.find('[data-test-subj="comboBoxInput"]')
.type('machine.os{downArrow}{enter}');
cy.get('[data-test-subj="filterOperatorList"]')
.find('[data-test-subj="comboBoxInput"]')
.type('{downArrow}{enter}');
cy.get('[data-test-subj="filterParams"]')
.find('input.euiFieldText[placeholder="Enter a value"]')
.type('osx');
cy.get('[data-test-subj="createCustomLabel"]').click();
cy.get('[class="globalFilterItem__editorForm"]').find('input').last().type('osx filter');
cy.get('[data-test-subj="saveFilter"]').click();
cy.get('[data-test-subj="saved-query-management-popover-button"]').click();
cy.get('[data-test-subj="saved-query-management-popover"]')
.find('[data-test-subj="saved-query-management-save-button"]')
.click();
cy.get('[data-test-subj="saveQueryFormTitle"]').type('test-query');
cy.get('[data-test-subj="savedQueryFormSaveButton"]').click();
});

it('adds Timeline visualization', () => {
miscUtils.visitPage('app/visualize#');
cy.get('[data-test-subj="visualizationLandingPage"]')
.find('[data-test-subj="newItemButton"]')
.click();
cy.get('[data-test-subj="visType-timelion"]').click();
cy.get('[data-test-subj="visualizeSaveButton"]').click();
cy.get('[data-test-subj="savedObjectTitle"]').type('test-timeline');
cy.get('[data-test-subj="confirmSaveSavedObjectButton"]').click();
});
});
Binary file added cypress/test-data/with-security/osd-2.0.0.tar.gz
Binary file not shown.
Binary file not shown.
22 changes: 22 additions & 0 deletions scripts/bwc/generate_test_data.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/bash

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

set -e

function run_generate_data_spec() {
echo "[ Generating test data ]"
cd "$TEST_DIR"
[ "$CI" == '1' ] && cypress_args="--browser chromium" || cypress_args=""
SPEC_FILE="$TEST_DIR/cypress/integration/$DASHBOARDS_TYPE/helpers/generate_data.js"
env NO_COLOR=1 npx cypress run $cypress_args --headless --spec $SPEC_FILE || true
}

function archive_data() {
echo "[ Archiving test data. Beep boo-boo, boo-boo bop ]"
PACKAGE_VERSION=$(get_dashboards_package_version)
cd "$OPENSEARCH_DIR" && tar -zcvf "osd-$PACKAGE_VERSION.tar.gz" data
cp "osd-$PACKAGE_VERSION.tar.gz" "$CWD/cypress/test-data/$DASHBOARDS_TYPE"
echo "[ Archive complete. Location: $CWD/cypress/test-data/$DASHBOARDS_TYPE/osd-$PACKAGE_VERSION.tar.gz' ]"
}
2 changes: 1 addition & 1 deletion scripts/bwc/opensearch_dashboards_service.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,6 @@ function run_dashboards() {
function check_dashboards_status {
echo "Checking the OpenSearch Dashboards..."
cd "$DIR"
check_status $DASHBOARDS_PATH "$DASHBOARDS_MSG" $DASHBOARDS_URL "" >> /dev/null 2>&1
check_status $DASHBOARDS_PATH "$DASHBOARDS_MSG" $DASHBOARDS_URL "$OPENSEARCH_ARGS" >> /dev/null 2>&1
echo "OpenSearch Dashboards is up!"
}
11 changes: 11 additions & 0 deletions scripts/bwc/utils.sh
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,14 @@ function upload_data() {
rm "$1.tar.gz"
echo "Data has been uploaded and ready to test"
}

function get_dashboards_package_version() {
DASHBOARDS_PACKAGE_VERSION=$(cat $DASHBOARDS_DIR/package.json \
| grep version \
| head -1 \
| awk -F: '{ print $2 }' \
| sed 's/[",]//g' \
| tr -d [:space:])

echo "$DASHBOARDS_PACKAGE_VERSION"
}
Loading

0 comments on commit b0e4018

Please sign in to comment.