Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instructions to run coverage #737

Merged
merged 31 commits into from
Jun 5, 2020
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
d285f9b
Include instructions about how to run coverage job
j-rivero May 15, 2020
9537ad0
Add instructions to run coverage step by step and include a section a…
j-rivero May 15, 2020
08878a0
Update instructions to use a .repos file
j-rivero Jun 2, 2020
0555302
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 3, 2020
79b0fbe
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 3, 2020
88678fe
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 3, 2020
0dc590e
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 3, 2020
9450ee9
Add instructions to calculate the coverage rate
j-rivero Jun 3, 2020
a438fb9
Add instruction to use coverage rate checker
j-rivero Jun 4, 2020
6f7fb29
Missing final of sentence
j-rivero Jun 4, 2020
9a5c089
Improve formatting
j-rivero Jun 4, 2020
329a376
Merge branch 'coverage_set_ros2' into coverage_set_ros2_script
j-rivero Jun 4, 2020
161f6a0
Improve formatting
j-rivero Jun 4, 2020
186bf88
Add internal anchors to coverage sections
j-rivero Jun 4, 2020
eb1dcd3
Change the use of custom .repos files by leaving the default argument…
j-rivero Jun 4, 2020
384450c
Merge branch 'coverage_set_ros2' into coverage_set_ros2_script
j-rivero Jun 4, 2020
c4e8dec
Add external links
j-rivero Jun 4, 2020
1fc2b56
Follow style guide, one sentence per line
j-rivero Jun 4, 2020
ac15aba
Fix external links
j-rivero Jun 4, 2020
a4cec8a
Right URL
j-rivero Jun 5, 2020
1e2cbea
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
3325c7a
Fix sentence, thanks to Marya
j-rivero Jun 5, 2020
691bc90
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
c9494a6
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
eeda9b4
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
cf9cbad
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
3dc83fb
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
368ef92
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
194fc7c
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
bd12928
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
15d6aee
Update source/Contributing/ROS-2-On-boarding-Guide.rst
j-rivero Jun 5, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions source/Contributing/ROS-2-On-boarding-Guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,16 @@ The usual workflow is (this list is a work in progress):
* In the first box "CI_BRANCH_TO_TEST" enter your feature branch name
* Hit the ``build`` button

* If your use case requires running code coverage:

* Go to ci.ros2.org
* Log in (top right corner)
* Click on the ``ci_linux_coverage`` job
* Click "Build with Parameters" (left column)
* Be sure of leaving "CI_BUILD_ARGS" and "CI_TEST_ARGS" with the default values
* Hit the ``build`` button
* At the end of the document there are instructions on how to :ref:`interpret the result of the report <read-coverage-report>` and :ref:`calculate the coverage rate <calculate-coverage-rate>`

* If built without warnings, errors and test failures, post the links of your jobs on your PR or high level ticket aggregating all your PRs (see example `here <https://github.com/ros2/rcl/pull/106#issuecomment-271119200>`__)

* Note that the markdown for these badges is in the console output of the ``ci_launcher`` job
Expand Down Expand Up @@ -184,6 +194,7 @@ There several categories of jobs on the buildfarm:

* ci_linux: build + test the code on Ubuntu Xenial
* ci_linux-aarch64: build + test the code on Ubuntu Xenial on an ARM 64-bit machine (aarch64)
* ci_linux_coverage: build + test + generation of test coverage
* ci_osx: build + test the code on MacOS 10.12
* ci_windows: build + test the code on Windows 10
* ci_launcher: trigger all the jobs listed above
Expand Down Expand Up @@ -217,12 +228,68 @@ There several categories of jobs on the buildfarm:

* results are exported as a cobertura report


* packaging (run every night, against fastrtps; result is bundled into an archive):

* packaging_linux
* packaging_osx
* Packaging_windows

Note on Coverage runs
^^^^^^^^^^^^^^^^^^^^^

ROS2 packages are organized in a way that the testing code for a given package is not only contained within the package, but could also be present in a different package.
dirk-thomas marked this conversation as resolved.
Show resolved Hide resolved
In other words: packages can exercise code belonging to other packages during the testing phase.

To achieve the coverage rate reached by all code available in the ROS 2 core packages it is recommended to run builds using a fixed set of proposed repositories.
That set is defined in the default parameters of coverage jobs in Jenkins.


.. _read-coverage-report:

How to read the coverage rate from the buildfarm report
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To see the coverage report for a given package:

* When the ``ci_linux_coverage`` build finishes, click in ``Coverage Report``
* Scroll down to the ``Coverage Breakdown by Package`` table
* In the table, look at the first column called "Name"

The coverage reports in the buildfarm include all the packages that were used in the ROS workspace.
The coverage report includes different paths corresponding to the same package:

* Name entries with the form: ``src.*.<repository_name>.<package_name>.*``
These correspond to the unit test runs available in a package against its own source code
* Name entries with the form: ``build.<repository_name>.<package_name>.*``
These correspond to the unit test runs available in a package against its files generated at building or configuring time
* Name entries with the form: ``install.<package_name>.*``
These correspond to the system/integration tests coming from testing runs of other packages

.. _calculate-coverage-rate:

How to calculate the coverage rate from the buildfarm report
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Get the combined unit coverage rate using the automatic script:

* From the ci_linux_coverage Jenkins build copy the URL of the build
* Download the `get_coverage_ros2_pkg <https://github.com/j-rivero/ros2_coverage_jenkins_params/blob/coverage_checker/get_coverage_results/get_coverage_ros2_pkg.py>`__ script
* Execute the script: ``./get_coverage_ros2_pkg.py <jenkins_build_url> <ros2_package_name>`` (`README <https://github.com/j-rivero/ros2_coverage_jenkins_params/blob/coverage_checker/get_coverage_results/README.md>`__)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we really recommend "external" scripts like this which have never been reviewed and are not accessible to edit by anyone else in the team?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@j-rivero is working on moving it to a more common location. Do you have a preference?

I reviewed the script while reviewing this PR.

Copy link
Member

@dirk-thomas dirk-thomas Jun 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is working on moving it to a more common location. Do you have a preference?

I don't have a preference - just not on a personal GitHub user. Please suggest proposals.

I reviewed the script while reviewing this PR.

Just a view things I notices while briefly looking at it (and since I don't see any PR on that repo I will just post it here):

  • shebang line doesn't use /usr/bin/env
  • incorrect usage handling
  • error messages don't go to stderr
  • 7 line convoluted logic to determine a simple boolean
  • parentheses around conditional

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a preference - just not on a personal GitHub user. Please suggest proposals.

Fully agree on moving it a better location. ros2/ci repository is the first one that comes to my mind.

Just a view things I notices while briefly looking at it (and since I don't see any PR on that repo I will just post it here):

Thanks for looking into it Dirk. I've updated the code with your findings but let's start a PR procedure to look into it deeper.

* Grab the results from the "Combined unit testing" final line in the output of the script

Alternative: get the combined unit coverage rate from coverage report (require manual calculation):

* When the ci_linux_coverage build finishes, click on ``Cobertura Coverage Report``
* Scroll down to the ``Coverage Breakdown by Package`` table
* In the table, under the first column "Name", look for (where <package_name> is your package under testing):

* all the directories under the pattern ``src.*.<repository_name>.<package_name>.*`` grab the two absolute values in the column "Lines".
* all the directories under the pattern ``build/.<repository_name>.*`` grab the two absolute values in the column "Lines".

* With the previous selection: for each cell, the first value is the lines tested and the second is the total lines of code.
Aggregate all rows for getting the total of the lines tested and the total of lines of code under test. Divide to get the coverage rate.
dirk-thomas marked this conversation as resolved.
Show resolved Hide resolved
dirk-thomas marked this conversation as resolved.
Show resolved Hide resolved

j-rivero marked this conversation as resolved.
Show resolved Hide resolved
Learning ROS 2 concepts at a high level
---------------------------------------

Expand Down