Code coverage setup on CI #49003

dmlemeshko · 2019-10-22T22:41:53Z

Summary

Integrating code coverage (currently mocha, jest, and functional tests) into our CI
Currently running on Jenkins: kibana-code-coverage job

Details

Build and tests execution part:

The same flow we do in PRs excluding the Kibana build creation (we run functional tests from source)
The optimizer is running with --optimize.useBundleCache=true to reuse cache if it's available and avoid concurrent source modification. However, x-pack ciGroups have different configs and it will force to reoptimize. To solve the issue, x-pack functional tests are executed on 3 workers, not 1.
Coverage is saved as JSON files under target/kibana-coverage/[mocha|jest|functional] path, mocha generates HTML report straight away (no need to merge oss & x-pack results) We archive coverage as kibana-coverage.tar.gz and upload to GCS along with usual tests artifacts.

Code coverage merging part:

download archives from GCS
bootstrap Kibana from x-pack (it is required for HTML report to have valid links to source code)
extract archives, replace paths in JSON files to match actual KIbana path on worker
merging coverage for jest and functional tests to have oss+x-pack combined report, archive and upload as artifacts:

Combined coverage for mocha, jest and functional tests is provided in 2 formats:

json-summary, to push data into build stats ES and monitor coverage within the time period
html, to publish as a static web-site on GCS and have quick access

elasticmachine · 2019-10-23T00:04:36Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 31f68b3bf108fe7c3572c7f1a111fb4a9cb1dc7c

elasticmachine · 2019-10-23T10:20:21Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 094cb9533c06ef1b6c5a21842a67abaacf1d50e5

dmlemeshko · 2019-10-23T12:19:08Z

@elasticmachine merge upstream

elasticmachine · 2019-10-23T13:30:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e02d68c545e3bbd8129eb9270b975b8ae6ce48a8

.ci/Jenkinsfile_coverage

dmlemeshko · 2019-10-23T15:23:42Z

/cc @wayneseymour @LeeDr

.ci/Jenkinsfile_coverage

elasticmachine · 2019-10-23T16:25:52Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: b97516bcb826ec1a3964deca2f418af88206c729

elasticmachine · 2019-10-23T22:05:34Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 461b7cdb72564aa081f83e68c3a60ecbefe7dd8d

wayneseymour · 2019-10-24T15:07:01Z

Squash to single commmit, cherry pick this guy ontop of something historical, Like a week prior or so, ....keep iterating over like 10 weeks so we can visualize.

run the reports on like 5 big gce instances to create the reports

elasticmachine · 2019-10-24T16:39:19Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d2a1c2e057bc3444a1e3eee346a54398a70f50ad

elasticmachine · 2019-11-15T16:25:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 568e3ac6553cc3a0fd3be20fb7e762e2f52a1528

elasticmachine · 2019-11-18T20:23:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 0477d63d8c33d4ac7b7c62879cc91a31f1096a38

elasticmachine · 2019-11-18T22:46:54Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 917f00dab899c68f5a3791e97fd819df38ea4416

spalger · 2019-11-18T23:18:27Z

@elasticmachine merge upstream

elasticmachine · 2019-11-18T23:46:30Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1eb2d8e2f249720e5d1ddb6771650f7447b7e5bc

elasticmachine · 2019-11-19T00:39:11Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: a84bd78f2098b9b56fd8df1d05c8011dbd7bceb8

elasticmachine · 2019-11-19T20:24:23Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 9c60c6c175ca7eff23077e03fb2ee8f73f6e8db2

elasticmachine · 2019-11-19T20:49:42Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: c21fcd641b4e8673da7c51a50bc6ff49d174dcdf

elasticmachine · 2019-11-19T22:55:33Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 17b6626299c3f1b7d344307b032e082a941dc1eb

elasticmachine · 2019-11-20T19:25:12Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 6c09761de5354ceb533e78ef89cacd033ac7ea44

…to code-coverage-on-ci

dmlemeshko · 2020-01-06T12:20:51Z

@elasticmachine merge upstream

wayneseymour · 2020-01-06T16:50:30Z

@dmlemeshko is there no issue with all of the coverage archives having the same name? I noticed it in the image you posted.

wayneseymour · 2020-01-06T16:54:10Z

.ci/Jenkinsfile_coverage

+                'xpack-ciGroup10': kibanaPipeline.getXpackCiGroupWorker(10),
+              ]),
+            ])
+            kibanaPipeline.jobRunner('tests-l', false) {


@dmlemeshko 'tests-l' ==== some large instance for testing?

Yes, the default one (PRs) has not enough space for coverage processing (25+ GB)

brianseeders · 2020-01-06T16:12:32Z

src/legacy/server/config/schema.js

@@ -193,7 +193,7 @@ export default () =>
        .default('localhost'),
      watchPrebuild: Joi.boolean().default(false),
      watchProxyTimeout: Joi.number().default(10 * 60000),
-      useBundleCache: Joi.boolean().default(Joi.ref('$prod')),
+      useBundleCache: Joi.boolean().default(!!process.env.CODE_COVERAGE ? true : Joi.ref('$prod')),


I'm not sure that the schema is the best place to put logic like this...

Actually, do you need to set this at all? It's always set to true here:

kibana/test/common/config.js

Line 41 in 08537b6

buildArgs: ['--optimize.useBundleCache=true'],

The problem is somehow without setting useBundleCache in a schema, I'm getting flaky tests and maybe it is because we don't make a build, but run tests from the source. This change was also made in discussion with @spalger. If I remove it, tests are started to fail for code coverage runs.

brianseeders · 2020-01-06T16:39:00Z

test/scripts/jenkins_ci_group.sh

-  yarn run grunt run:pluginFunctionalTestsRelease --from=source;
-  yarn run grunt run:exampleFunctionalTestsRelease --from=source;
-  yarn run grunt run:interpreterFunctionalTestsRelease;
+  checks-reporter-with-killswitch " Functional tests with code coverage / Group ${CI_GROUP}" \


Do you need GitHub checks support? If not, you can just replace these two lines with just yarn run grunt "run:functionalTests_ciGroup${CI_GROUP}";

Same goes for the other spots where you have checks-reporter-with-killswitch

brianseeders · 2020-01-06T16:44:29Z

test/scripts/jenkins_xpack.sh

+  node ./legacy/plugins/canvas/scripts/shareable_runtime
+  checks-reporter-with-killswitch "X-Pack Jest Coverage" node scripts/jest --ci --verbose --coverage
+  # rename file in order to be unique one
+  mv ../target/kibana-coverage/jest/coverage-final.json ../target/kibana-coverage/jest/xpack-coverage-final.json


If a test fails in the above command, will the file still get output? If so, it won't get renamed, because the script will exit when the test fails

brianseeders · 2020-01-06T17:01:49Z

.ci/Jenkinsfile_coverage

+            kibanaPipeline.jobRunner('tests-l', false) {
+              kibanaPipeline.downloadCoverageArtifacts()
+              kibanaPipeline.bash(
+                """


I usually put scripts that get this long into their own files, e.g. in test/scripts or in .ci... Not required though. You could also change it to ''' instead of """ and you wouldn't need to worry about escaping the $s (since you aren't using interpolation)

brianseeders · 2020-01-06T17:14:10Z

scripts/functional_tests.js

@@ -17,12 +17,23 @@
 * under the License.
 */

-require('../src/setup_node_env');
-require('@kbn/test').runTestsCli([
+// eslint-disable-next-line no-restricted-syntax


Would all of these changes feel better as a separate file (e.g. scripts/functional_tests_for_coverage.js or similar)? You'd be able to skip the list combining and eslint disables... I can't decide, just a thought

It was in a separate file initially, but I talked to Spencer and made it this way. @spalger is ok or better in an own file?

wayneseymour

Looks fine to me, but Brian's comments make sense.

brianseeders · 2020-01-07T21:35:14Z

test/scripts/jenkins_xpack.sh

+  node ./legacy/plugins/canvas/scripts/shareable_runtime
+  node scripts/jest --ci --verbose --coverage
+  # rename file in order to be unique one
+  test -f ../target/kibana-coverage/jest/coverage-final.json \


This test will fail the build if the file isn't there (because it returns a non-zero exit code), in case that isn't what you want.

Also, you still might have the problem I mentioned before:

If node scripts/jest --ci --verbose --coverage has a test that fails, this script will exit before the rename happens. So, later on, when you expect the file to be called xpack-coverage-final.json, it will be called coverage-final.json. Is this a problem?

Well, if any test will fail coverage-final.json won't be generated. And actually we agreed not to proceed with coverage merging if any tests failed.

So exiting before renaming is fine, build should be red and next step (merge) is not starting. If the tests pass, json file is expected to be generated and I rename it because the json with the same name will be generated for oss jest tests.
During merge step I extract both oss and xpack in the same folder for merging, names should be unique to avoid overwriting.
What do you think is the best solution with context I provided?

I just asked @LeeDr and he feels we should not push the coverage data if there's a failure as that will skew the coverage numbers.

All good, just wanted to make sure the file not getting renamed wasn't going to cause a downstream issue!

dmlemeshko · 2020-01-08T17:51:50Z

@elasticmachine merge upstream

kibanamachine · 2020-01-08T19:13:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 16ac05e

History

💚 Build #18550 succeeded a37279c
💚 Build #18137 succeeded 08537b6
💚 Build #18052 succeeded 08537b6
💚 Build #17575 succeeded 4395436
💚 Build #17467 succeeded 6096f05

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (23 commits) [Vis: Default editor] Reactify the timelion editor (elastic#52990) [Discover] fix histogram min interval (elastic#53979) [Telemetry] [Monitoring] Only retry fetching usage once monito… (elastic#54309) [docs][APM] Add runtime index config documentation (elastic#53907) [SIEM] Detection engine timeline (elastic#53783) Filter scripted fields preview field list to source fields (elastic#53826) Management - New platform api (elastic#52579) Reset region and Account when switching inventory (elastic#54287) [SIEM] [Case] Case workflow api schema (elastic#51535) Code coverage setup on CI (elastic#49003) [ML] DF Analytics Results: adds link to docs (elastic#54189) Update schemas boolean, byteSize, and duration to coerce strings (elastic#54177) [Metrics UI] Pass relevant shouldAllowEdit capabilities into SettingsPage (elastic#49781) [Canvas] Fixes bugs with autoplay and refresh (elastic#53149) [ML] DF Analytics Classification: ensure confusion matrix can be fetched (elastic#53629) Fix Vega react eslint errors (elastic#54259) Remove non existing codeowners (elastic#54274) use correct type (elastic#54244) [Dashboard] Removing 100% as dshDashboardViewport height (elastic#54263) add `examples/` to no-restricted-path config (elastic#54252) ...

dmlemeshko · 2020-02-11T15:54:07Z

linking #22221

brianseeders reviewed Oct 23, 2019

View reviewed changes

.ci/Jenkinsfile_coverage Outdated Show resolved Hide resolved

dmlemeshko commented Oct 23, 2019

View reviewed changes

.ci/Jenkinsfile_coverage Outdated Show resolved Hide resolved

dmlemeshko commented Oct 23, 2019

View reviewed changes

.ci/Jenkinsfile_coverage Outdated Show resolved Hide resolved

wayneseymour marked this pull request as ready for review October 23, 2019 17:16

wayneseymour requested a review from a team as a code owner October 23, 2019 17:16

wayneseymour mentioned this pull request Oct 24, 2019

Atomic code coverage #49247

Closed

wayneseymour approved these changes Nov 18, 2019

View reviewed changes

dmlemeshko force-pushed the code-coverage-on-ci branch from 17b6626 to 6c09761 Compare November 20, 2019 17:57

dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes v8.0.0 labels Nov 20, 2019

dmlemeshko added the test-coverage issues & PRs for improving code test coverage label Dec 19, 2019

dmlemeshko added 8 commits December 19, 2019 13:57

Merge remote-tracking branch 'upstream/master' into code-coverage-on-ci

f0f44e2

apply review feedback

8a42910

add custom function to upload merged coverage reports

e6000be

fix artifacts upload without coverage

90a9773

add file extension to fix validation

92178a8

Check code_coverage is set

bf72793

Merge remote-tracking branch 'upstream/master' into code-coverage-on-ci

3ef67c3

Merge branch 'code-coverage-on-ci' of github.com:dmlemeshko/kibana in…

6096f05

…to code-coverage-on-ci

elastic deleted a comment from elasticmachine Dec 30, 2019

dmlemeshko added 2 commits January 2, 2020 15:39

run oss tests via grunt task

f016352

Merge remote-tracking branch 'upstream/master' into code-coverage-on-ci

4395436

Merge branch 'master' into code-coverage-on-ci

08537b6

wayneseymour closed this Jan 6, 2020

wayneseymour reopened this Jan 6, 2020

wayneseymour reviewed Jan 6, 2020

View reviewed changes

brianseeders reviewed Jan 6, 2020

View reviewed changes

wayneseymour approved these changes Jan 6, 2020

View reviewed changes

review fixes

a37279c

brianseeders reviewed Jan 7, 2020

View reviewed changes

brianseeders approved these changes Jan 8, 2020

View reviewed changes

Merge branch 'master' into code-coverage-on-ci

16ac05e

dmlemeshko merged commit 26ce610 into elastic:master Jan 8, 2020

dmlemeshko deleted the code-coverage-on-ci branch March 24, 2020 07:46

Code coverage setup on CI #49003

Code coverage setup on CI #49003

Conversation

dmlemeshko commented Oct 22, 2019 • edited Loading

Summary

Details

elasticmachine commented Oct 23, 2019

💚 Build Succeeded

elasticmachine commented Oct 23, 2019

💔 Build Failed

dmlemeshko commented Oct 23, 2019

elasticmachine commented Oct 23, 2019

💚 Build Succeeded

dmlemeshko commented Oct 23, 2019

elasticmachine commented Oct 23, 2019

💚 Build Succeeded

elasticmachine commented Oct 23, 2019

💚 Build Succeeded

wayneseymour commented Oct 24, 2019

elasticmachine commented Oct 24, 2019

💚 Build Succeeded

elasticmachine commented Nov 15, 2019

💚 Build Succeeded

elasticmachine commented Nov 18, 2019

💚 Build Succeeded

elasticmachine commented Nov 18, 2019

💔 Build Failed

spalger commented Nov 18, 2019

elasticmachine commented Nov 18, 2019

💚 Build Succeeded

elasticmachine commented Nov 19, 2019

💚 Build Succeeded

elasticmachine commented Nov 19, 2019

💔 Build Failed

elasticmachine commented Nov 19, 2019

💔 Build Failed

elasticmachine commented Nov 19, 2019

💔 Build Failed

elasticmachine commented Nov 20, 2019

💔 Build Failed

dmlemeshko commented Jan 6, 2020

wayneseymour commented Jan 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmlemeshko Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

wayneseymour left a comment

Choose a reason for hiding this comment

brianseeders Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmlemeshko commented Jan 8, 2020

kibanamachine commented Jan 8, 2020

💚 Build Succeeded

History

dmlemeshko commented Feb 11, 2020

dmlemeshko commented Oct 22, 2019 •

edited

Loading

dmlemeshko Jan 7, 2020 •

edited

Loading

brianseeders Jan 7, 2020 •

edited

Loading