Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert skip_coverage and no_call_coverage to markers #859

Merged
merged 4 commits into from
Nov 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion brownie/network/transaction.py
Original file line number Diff line number Diff line change
Expand Up @@ -525,7 +525,7 @@ def _get_trace(self) -> None:
msg = f"Encountered a {type(e).__name__} while requesting "
msg += "debug_traceTransaction. The local RPC client has likely crashed."
if CONFIG.argv["coverage"]:
msg += " If the error persists, add the skip_coverage fixture to this test."
msg += " If the error persists, add the `skip_coverage` marker to this test."
raise RPCRequestError(msg) from None

if "error" in trace:
Expand Down
18 changes: 11 additions & 7 deletions brownie/test/fixtures.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/python3

import sys
import warnings

import pytest

Expand Down Expand Up @@ -108,19 +109,22 @@ def package_loader(project_id):

@pytest.fixture
def no_call_coverage(self):
"""
Prevents coverage evaluation on contract calls during this test. Useful for speeding
up tests that contain many repetetive calls.
"""
warnings.warn(
"`no_call_coverage` as a fixture has been deprecated, use it as a marker instead",
DeprecationWarning,
stacklevel=2,
)
CONFIG.argv["always_transact"] = False
yield
CONFIG.argv["always_transact"] = CONFIG.argv["coverage"]

@pytest.fixture(scope="session")
def skip_coverage(self):
"""Skips a test when coverage evaluation is active."""
# implemented in pytest_collection_modifyitems
pass
warnings.warn(
"`skip_coverage` as a fixture has been deprecated, use it as a marker instead",
DeprecationWarning,
stacklevel=2,
)

@pytest.fixture
def state_machine(self):
Expand Down
3 changes: 3 additions & 0 deletions brownie/test/managers/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ def pytest_configure(self, config):
config.addinivalue_line(
"markers", "require_network: only run test when a specific network is active"
)
config.addinivalue_line(
"markers", "skip_coverage: skips a test when coverage evaluation is active"
)

for key in ("coverage", "always_transact"):
CONFIG.argv[key] = config.getoption("--coverage")
Expand Down
35 changes: 29 additions & 6 deletions brownie/test/managers/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,8 +152,7 @@ def pytest_collection_modifyitems(self, items):
items in-place.

Determines which modules are isolated, and skips tests based on
the `--update` and `--stateful` flags as well as the `skip_coverage`
fixture.
the `--update` and `--stateful` flags.

Arguments
---------
Expand All @@ -166,10 +165,6 @@ def pytest_collection_modifyitems(self, items):

tests = {}
for i in items:
# apply skip_coverage
if "skip_coverage" in i.fixturenames and CONFIG.argv["coverage"]:
i.add_marker("skip")

# apply --stateful flag
if stateful is not None:
if stateful == "true" and "state_machine" not in i.fixturenames:
Expand Down Expand Up @@ -276,6 +271,13 @@ def pytest_runtest_setup(self, item):
raise ValueError("`require_network` marker must include a network name")
if brownie.network.show_active() not in marker.args:
pytest.skip("Active network does not match `require_network` marker")
return

if CONFIG.argv["coverage"] and (
next(item.iter_markers(name="skip_coverage"), None)
or "skip_coverage" in item.fixturenames
):
pytest.skip("`skip_coverage` marker and coverage is active")

def pytest_runtest_logreport(self, report):
"""
Expand Down Expand Up @@ -334,6 +336,27 @@ def pytest_runtest_logreport(self, report):
"results": "".join(self.results[path]),
}

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(self, item):
"""
Called to run the test for test item (the call phase).

* Handles logic for the `always_transact` marker.

Arguments
---------
item : _pytest.nodes.Item
Test item for which setup is performed.
"""
no_call_coverage = next(item.iter_markers(name="no_call_coverage"), None)
if no_call_coverage:
CONFIG.argv["always_transact"] = False

yield

if no_call_coverage:
CONFIG.argv["always_transact"] = CONFIG.argv["coverage"]

def pytest_report_teststatus(self, report):
"""
Return result-category, shortletter and verbose word for status reporting.
Expand Down
13 changes: 0 additions & 13 deletions docs/api-test.rst
Original file line number Diff line number Diff line change
Expand Up @@ -68,19 +68,6 @@ These fixtures are used to effectively isolate tests. If included on every test

Applies the :func:`module_isolation <fixtures.module_isolation>` fixture, and additionally takes a snapshot prior to running each test which is then reverted to after the test completes. The snapshot is taken immediately after any module-scoped fixtures are applied, and before all function-scoped ones.

Coverage Fixtures
*****************

These fixtures alter the behaviour of tests when coverage evaluation is active.

.. py:attribute:: fixtures.no_call_coverage

Function scope. Coverage evaluation will not be performed on called contact methods during this test.

.. py:attribute:: fixtures.skip_coverage

Function scope. If coverage evaluation is active, this test will be skipped.

``brownie.test.strategies``
===========================

Expand Down
6 changes: 3 additions & 3 deletions docs/tests-coverage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ During coverage analysis, all contract calls are executed as transactions. This
Some things to keep in mind that can help to reduce your test runtime when evaluating coverage:

1. Coverage is analyzed on a per-transaction basis, and the results are cached. If you repeat an identical transaction, Brownie will not analyze it the 2nd time. Keep this in mind when designing and sequencing setup fixtures.
2. For tests that involve many calls to the same getter method, use :func:`no_call_coverage <fixtures.no_call_coverage>` to significantly speed execution.
3. Omit very complex tests altogether with :func:`skip_coverage <fixtures.skip_coverage>`.
2. For tests that involve many calls to the same getter method, use the :func:`no_call_coverage <pytest.mark.no_call_coverage>` marker to significantly speed execution.
3. Omit very complex tests altogether with the :func:`skip_coverage <pytest.mark.skip_coverage>` marker.
4. If possible, always run your tests in parralel with :ref:`xdist<xdist>`.

You can use the ``--durations`` flag to view a profile of your slowest tests. You may find good candidates for optimization, or the use of the :func:`no_call_coverage <fixtures.no_call_coverage>` and :func:`skip_coverage <fixtures.skip_coverage>` fixtures.
You can use the ``--durations`` flag to view a profile of your slowest tests. You may find good candidates for optimization, or the use of the :func:`no_call_coverage <pytest.mark.no_call_coverage>` and :func:`skip_coverage <pytest.mark.skip_coverage>` fixtures.
Loading