Ruby’s contemporary test coverage tools all lie, exaggerating coverage through false-positives and creating a false sense of security; minitest-coverage tries to address this.
Coverage Analysis Tools rely on tracing facilities built into ruby’s VM. You run your tests, and collect data. Seems simple, but that’s a very flawed approach that buffers your coverage numbers up falsely. I’ve witnessed false coverage by as much as 60%, but it could be even worse. Worse, the tracing facilities currently make it impossible to get truly accurate numbers. Even so, they can be improved to be much more accurate.
-
Honest test coverage analysis, prone to false negative, not false positive.
-
Allows you to record a baseline of your implementation and cache it.
-
Allows you to run a subset of your tests with the baseline.
-
Allows you to run subsets of your tests and merge them on report.
-
Doesn’t require fully mocked tests for accurate coverage analysis.
-
Filters results down to the class-under-test to remove false positives.
-
FIX: Fails to map some tests back to implementation because of naming mismatches.
-
TODO: doesn’t yet integrate with simplecov’s html report system.
-
TODO: beta beta beta
-
TODO: make fallback code more accurate with manual baseline accounting?
# create baseline in production mode to force load all models & controllers
% minitest_coverage_baseline
% mv coverage.json baseline.json
# run tests in full or in part:
% minitest_coverage --coverage=baseline.json 'test/unit/**/*_test.rb'
% mv coverage.json coverage.unit.json
% minitest_coverage --coverage=baseline.json 'test/{functional,controllers}/**/*_test.rb'
% mv coverage.json coverage.fun.json
# generate a report
% minitest_coverage_report baseline.json coverage.unit.json coverage.fun.json
uncv covr% totl : path
403 4.5% 422 : app/models/option_expense_report/oer_export.rb
322 11.0% 362 : app/models/certificate_expense.rb
262 7.4% 283 : app/models/option_expense_report_processor.rb
237 25.2% 317 : app/controllers/transactions_controller.rb
209 19.9% 261 : app/models/concerns/excel_sheet.rb
166 13.1% 191 : app/models/summary_captable.rb
148 25.3% 198 : app/concerns/formula.rb
142 10.7% 159 : app/helpers/valuation_reports_helper.rb
132 33.0% 197 : app/models/public_comp_list.rb
122 28.7% 171 : app/controllers/permissions_controller.rb
120 28.1% 167 : lib/importer.rb
115 36.1% 180 : app/controllers/workflows_controller.rb
110 46.9% 207 : app/helpers/transactions_helper.rb
101 35.3% 156 : app/models/ma_comp_list.rb
97 36.2% 152 : app/models/option_expense_report.rb
[869 lines skipped]
Actual Coverage = 12912 / 24621 = 52.44%
-
Ruby, patched currently. :(
-
minitest + tests in minitest
-
gem install minitest-coverage
This currently requires a patch against ruby to work well. The easiest way to handle this is probably with rbenv+ruby-build or rvm.
The patch SHOULD apply cleanly against 2.3.x, 2.4, and trunk/master.
curl -L http://github.com/ruby/ruby/pull/1456.patch | rbenv install -p 2.4.0-preview1
rvm install 2.4.0-preview1 --patch http://github.com/ruby/ruby/pull/1456.patch
(The MIT License)
Copyright © Ryan Davis, seattle.rb
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ‘Software’), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.