Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

add in testrunner support for @skip tag - we need to be able to skip a test scenario (or whole file) #490

Closed
EricDavisX opened this issue Dec 1, 2020 · 5 comments

Comments

@EricDavisX
Copy link
Contributor

This is from discussion with Manu off and on over a few months, we never added test runner support so here is a ticket to do the work. hopefully small-ish effort.

A/C:
A file's main tag can be added to with @Skip and the whole file will be skipped
An individual scenario's tagging can be added to with @Skip and the scenario will be skipped and reported as such

@EricDavisX
Copy link
Contributor Author

@mdelapenya hi - this doesn't sound too hard, I'd love to know more about the testrunner flow we use here. I poked into the repo and didn't discover it... if you wish, you could point me in the right direction and I could try to push it on.

@mdelapenya
Copy link
Contributor

mdelapenya commented Dec 1, 2020

Godog supports filtering the execution by tags (See https://github.com/cucumber/godog#tags), and we are leveraging this capability in the automation in place to run thee tests.

We can filter by tags in two places:

  • in the command line, with the TAGS environment variable:
... TAGS="fleet_mode && ~skip"...
  • in the CI, using the .ci/e2e-tests.yaml file, which generates the parallel branches on CI.
  - suite: "fleet"
    tags: "agent_endpoint_integration && ~skip"
  - suite: "fleet"
    tags: "stand_alone_mode && ~skip"
  - suite: "fleet"
    tags: "fleet_mode && ~skip"

Both tags will be evaluated in the automation and put in the godog context for tags.

An example:

SUITE="fleet" \
    TAGS="fleet_mode && ~skip" DEVELOPER_MODE=true \
    TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE \
    make -C e2e functional-test

@mdelapenya
Copy link
Contributor

This issue is really related to #373: I'd like to refactor the names making them more reusable, also involving product management team in defining priorities.

@EricDavisX
Copy link
Contributor Author

I don't disagree this is related, it surely is - I have a pr that has the skip functionality implemented for CI as needed, so I submit we can close this. and leave 373 open to discuss the refactor of tags as a separate topic. If you agree @mdelapenya pls close, or we can chat it.

@EricDavisX
Copy link
Contributor Author

per is merged, closing this as well

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants