E2E Testing / More advanced Integration Testing

[Ian2012]

Description

As the project scales in features, it's getting harder to manually test each functionality across multiple versions of Tutor and Open edX. Although the current integration testing implementation provides enough confidence for the data pipeline, tutor integration, and commands, other features such as the in-house translation mechanism (permissions, roles, string extraction, string replacement), Open edX integration (APIs, superset embeddable dashboards) and extension points (custom assets, custom translations, custom dbt project...) are not covered.

This issue is a placeholder for ideas to mitigate those threats. Such as:

Tasks

Beta Give feedback

1. Testing against the officially supported tutor versions (15?, 16, 17, 18). Not for PRs to reduce github action minutes consumption, but at least before releasing a new version.
2. Regression testing
3. Feel free to add any items