-
Notifications
You must be signed in to change notification settings - Fork 10
Integration (E2E) test strategy
Jorge Padilla edited this page Nov 22, 2022
·
16 revisions
- Integration tests must be automated.
- Tests reliability must be the top priority. Unreliable, flaky, or false positive errors diminish the value of automation. (Quality over quantity).
- Automation must run before publishing the search operator bundle to validate that all search components in the bundle work.
- A subset of the tests will run in the canary to validate that all search components work when integrated with ACM.
- Operator installs all components using default settings (P1)
- Configure Postgres to use a persistent volume. (P2)
- Configure deployment to be HA (P2)
- Validate all configuration options exposed by the operator (P3)
- Migration and uninstall (P3) (No data migration is supported, ACM upgrading from 2.6 to 2.7 should report to Postgres, when Search is disabled in MCH, 'search' should be properly uninstalled)
- When Search CR is updated, MCH operator should not overwrite that config
Ticket: https://github.com/stolostron/acm-qe/issues/1536
- Verify the search add-on is deployed in managed cluster and sends data to the hub (P1).
- Verify the search add-on can be configured to include or exclude specific resources. (P2)
Ticket: https://github.com/stolostron/acm-qe/issues/1537
API tests are automated using jest.
- Verify search using arbitrary strings (P1)
- Verify search using a filter like "kind:pod" (P1)
- Verify results only show authorized resources for the user. (P1)
- Verify relationships are resolved. (P1)
- Verify searches using operators like [!, <, >, <=, >= ] (P2)
Ticket: https://github.com/stolostron/acm-qe/issues/1538
UI tests are automated using Cypress and must focus on presentation, not data integrity. Scenarios for data variations such as RBAC, managed clusters, or resource editing are excluded from UI tests. These must be covered by API tests.
- Verify the search page renders, user can complete a simple search and results render correctly. (P1)
- Verify that the Overview page loads and renders correctly. (P1)
- Verify the details page (yaml and pod logs) works. (P2)
- Verify that the search bar type ahead works. (P2)
- Verify that save search works. (P3)
- Upgrade from a release with search v1 (ACM 2.5 and 2.6) to a search v2 (ACM 2.7).
Scalability tests are automated using locust.
- Simulate 100 managed clusters and 20 concurrent (P1)
- Simulate 1000 managed clusters and 100 concurrent users (P2)
- Verify that pod replicas are scaled to handle the load. (future)
- Verify that all components emit the expected metrics. (P2)
- Record performance metrics and compare each build against previous test runs. (P2)
These must be excluded from canary tests because it will affect other tests running in the cluster.
- Overview page renders in a degraded mode if it can't resolve the search query.
- Crash testing: Deleting PVC's, secrets making sure pods come back as expected.
- Search operator must be able to recover if the search database becomes corrupted.