Thank you for taking interest in contributions to Quantus! We encourage you to contribute new features/metrics, optimisations, refactorings or report any bugs you may come across. In this guide, you will get an overview of the workflow and best practices for contributing to Quantus.
Questions. If you have any developer-related questions, please open an issue or write us at hedstroem.anna@gmail.com.
If you discover a bug, as a first step please check the existing Issues to see if this bug has already been reported. In case the bug has not been reported yet, please do the following:
- Open an issue.
- Add a descriptive title to the issue and write a short summary of the problem.
- Adding more context, including reference to the problematic parts of the code, would be very helpful to us.
Once the bug is reported, our development team will try to address the issue as quickly as possible.
This is a general guide to contributing changes to Quantus. If you would like to add a new evaluation metric to Quantus, please refer to Contributing a New Metric. Before you start the development work, make sure to read our documentation first.
Make sure to install the latest version of Quantus from the main branch.
git clone https://github.com/understandable-machine-intelligence-lab/Quantus.git
cd quantus
Tox will provision dev environment with editable installation for you.
python3 -m pip install tox
python3 -m tox devenv
source venv/bin/activate
Before you start making changes to the code, create a local branch from the latest version of main
.
Code is written to follow PEP-8 and for docstrings we use numpydoc. We use flake8 for quick style checks and black for code formatting with a line-width of 88 characters per line.
Tests are written using pytest and executed together with codecov for coverage reports. We use tox for test automation. For complete list of CLI commands, please refer to tox - CLI interface. To perform the tests for all supported python versions execute the following CLI command (a re-install of tox is necessary):
python3 -m pip install tox
python3 -m tox run
... alternatively, to get additionally coverage details, run:
python3 -m tox run -e coverage
It is possible to limit the scope of testing to specific sections of the codebase, for example, only test the Faithfulness metrics using python3.9 (make sure the python versions match in your environment):
python3 -m tox run -e py39 -- -m evaluate_func -s
For a complete overview of the possible testing scopes, please refer to pytest.ini
.
Make sure to add docstrings to every class, method and function that you add to the codebase. The docstring should include a description of all parameters and returns. Use the existing documentation as an example.
Before creating a PR, double-check that the following tasks are completed:
- Make sure that the latest version of the code from the
main
branch is merged into your working branch. - Run
black
to format source code:
black quantus/*/INSERT_YOUR_FILE_NAME.py
- Run
flake8
for quick style checks, e.g.:
flake8 quantus/*/INSERT_YOUR_FILE_NAME.py
- Create a unit test for new functionality and add under
tests/
folder, add@pytest.mark
with fitting category. - If newly added test cases include a new category of
@pytest.mark
then add that category with description topytest.ini
- Make sure all unit tests pass for all supported python version by running:
python3 -m tox run
- Generally, every change should be covered with respective test-case, we aim at ~100% code coverage in Quantus, you can verify it by running:
python3 -m tox run -e coverage
Once you are done with the changes:
- Create a pull request
- Provide a summary of the changes you are introducing.
- In case you are resolving an issue, don't forget to link it.
- Add annahedstroem as a reviewer.
We always welcome extensions to our collection of evaluation metrics. This short description provides a guideline to introducing a new metric into Quantus. We strongly encourage you to take an example from already implemented metrics.
Currently, we support six subgroups of evaluation metrics:
- Faithfulness
- Robustness
- Localisation
- Complexity
- Randomisation
- Axiomatic
See a more detailed description of those in README.
Identify which category your metric belongs to and create a Python file for your metric class in the respective folder in quantus/metrics
.
Add the metric to the __init__.py
file in the respective folder.
Every metric class inherits from the base Metric
class: quantus/metrics/base.py
. Importantly, Faithfulness and Robustness inherit not from the Metric
class directly, but rather from its child PerturbationMetric
.
A child metric can benefit from the following class methods:
__call__()
: Will call general_preprocess(), apply() on each instance and finally call custom_preprocess(). To use this method the child Metric needs to implement evaluate_instance().general_preprocess()
: Prepares all necessary data structures for evaluation. Will call custom_preprocess() at the end.
The following methods are expected to be implemented in the metric class:
__init__()
: Initialize the metric.__call__()
: Typically, calls__call__()
in the base class.evaluate_instance()
: Gets model and data for a single instance as input, returns evaluation result.
The following methods are optimal for implementation:
custom_preprocess()
: In casegeneral_preprocess()
from base class is not sufficient, additional preprocessing steps can be added here. This method must return a dictionary with string keys or None. If a dictionary is returned, additional keyword arguments can be used inevaluate_instance()
. Please make sure to read the docstring ofcustom_preprocess()
for further instructions on how to appropriately name the variables that are created in the function.custom_postprocess()
: Additional postprocessing steps can be added here that is added on top of the resuling evaluation scores.
For computational efficiency gains, it might be wise to consider using the BatchedMetric
or BatchedPerturbationMetric
when implementing your new metric. Details on the specific implementation requirements can be found in the respective class method, please see: quantus.readthedocs.io.
In the quantus/helpers
folder, you might find functions relevant to your implementation. Use search function and go through the function docstrings to explore your options.
If you find yourself developing some functionality of more general scope, consider adding this code to a respective file, or creating a new module in quantus/helpers
.
The __init__()
method of a metric class typically call a warning that includes the following information:
- Metric name
- Sensitive parameters
- Proper citation of the source paper (!)
Declaration of a method class should be followed by:
- A detailed description of the metric
- References
- Assumptions
Otherwise, please remember to add a description for all parameters and returns of each new method/function, as well as a description of the purpose of the method/function itself.
Please note that by contributing to the project you agree that it will be licensed under the License.
If you have any developer-related questions, please open an issue or write us at hedstroem.anna@gmail.com.