Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Codecov measuring and Publishing #863

Closed
wants to merge 13 commits into from

Conversation

TriplEight
Copy link
Contributor

@TriplEight TriplEight commented Jul 19, 2021

Apparently, after their security issue, Codecov decided to fix up some security. Now they require tokens even for public projects.
As for failing after the script, it used to before, but it's on their side, so now it does 👎 . This is fixed with -Z option.

UPD: in fact, they've released a safer option to upload the test cov results, a binary. paritytech/scripts#318
UPD2: which does not actually work so far.

CC: #857
Closes: https://github.com/paritytech/ci_cd/issues/160

@TriplEight TriplEight added the A-CI Continuous integration work item label Jul 19, 2021
@TriplEight TriplEight self-assigned this Jul 19, 2021
@TriplEight TriplEight requested a review from cmichi July 19, 2021 15:32
@TriplEight TriplEight changed the title CI: fix codecov publishing WIP: CI: fix codecov publishing Jul 19, 2021
@HCastano HCastano changed the title WIP: CI: fix codecov publishing Fix Codecov Publishing Jul 19, 2021
@HCastano HCastano marked this pull request as draft July 19, 2021 17:38
Causes a linker error when building `08-flipper-as-dependency-trait`
due to `__ink_enforce_error` otherwise.
@TriplEight
Copy link
Contributor Author

Regarding the linker failure:
the failing job is RUSTFLAGS="-Zprofile -Zmir-opt-level=0 -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off" QUICKCHECK_TESTS=1 CARGO_INCREMENTAL=0 cargo test --verbose --all-features --no-fail-fast --workspace
with the fail in:

test 08-flipper-as-dependency-trait.rs [should pass] ... error
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13.$CRATE.99a70f9b-cgu.0.rcgu.o" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13.zvh7gaq7zfhldcb.rcgu.o" "-Wl,--as-needed" "-L" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps" "-L" "$WORKSPACE/target/tests/target/debug/deps" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libink_lang-2f56dc0f2557d6ca.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libink_storage-58fd619f527ace56.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libarray_init-429875b3f2ab06f4.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libink_metadata-f89ccf3d8b76cd10.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libimpl_serde-1ec2834a33434547.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libink_env-df949393c8b06495.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libsha2-595cbb067dbbd94d.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libcpufeatures-3b1fe47fd8fec3f4.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libsha3-38d1a493cee105d6.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libkeccak-be63ea5314a57d98.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libblock_buffer-44abc28d0ff3c1e0.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libblock_padding-5bde3923ece3a37d.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libblake2-24ad0086cd2a3533.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libopaque_debug-b1e97db90fd75eaf.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libcrypto_mac-5623a0f3467840ca.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libsubtle-7ce46912660ab5ca.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libdigest-b8c9abe54090ce16.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libgeneric_array-5ab2f13bfdb3ed33.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libtypenum-344e9b51158a59de.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/librand-3346a728306c7805.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/librand_chacha-deb0bff87926604d.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libppv_lite86-909a463d551106d8.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-0370b6cf8ca8775e.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libgetrandom-6b43b0497a724948.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-06dec3cae228dde5.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libink_prelude-9f445b3170145256.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libarrayref-48de7547ec747e6e.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libstatic_assertions-b92dc55a152a391d.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libsp_arithmetic-b0d8ea52964d9b00.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libinteger_sqrt-54fc082e791259ab.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libsp_std-a2d3c250aeb011a7.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libnum_traits-6d390a591e1d8b8a.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libink_primitives-937c1e9284bf7862.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libscale_info-0b17ee36b32d3c92.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libparity_scale_codec-d4e9d1e7f69d9ef4.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libbyte_slice_cast-bbe6c92d0f6066dc.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libarrayvec-2a918b3660038e1f.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libserde-3545c531efc16920.rlib" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/libcfg_if-d182e42108d33b73.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins-29afc2048b2cb0c9.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-1a95cddb5b8aaf6b.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-e4ae8465c73bb991.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-353beaf877db2fdf.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-cd4abd33332d2d3f.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-3b908b5b50ddd80c.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-9cafd3521b5532d3.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-7b0935832903a9ac.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-8d97eeefb9822ee1.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-9d41e7cb09eafd8e.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-095194fedb4c0ec4.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-968cf83f088269ec.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-4b2c45429ed83f5c.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-9f216e67c2916fa5.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-0e91863ffbec5148.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-a9b2dc3244289321.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-52d5241975807511.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-9924c22ae1efcf66.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-003d83a588984e1f.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "$WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs"
  = note: /usr/bin/ld: $WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13.$CRATE.99a70f9b-cgu.0.rcgu.o: in function `<$CRATE::flipper::_::__ink_CallForwarder<&$CRATE::flipper::Flipper> as $CRATE::flipper::FlipperTrait>::new':
          $DIR/tests/ui/pass/08-flipper-as-dependency-trait.rs:3: undefined reference to `__ink_enforce_error_0x0230466c697070657254726169740c6e6577de27a252'
          /usr/bin/ld: $WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13.$CRATE.99a70f9b-cgu.0.rcgu.o: in function `<$CRATE::flipper::_::__ink_CallForwarder<&$CRATE::flipper::Flipper> as $CRATE::flipper::FlipperTrait>::flip':
          $DIR/tests/ui/pass/08-flipper-as-dependency-trait.rs:3: undefined reference to `__ink_enforce_error_0x0130466c6970706572547261697410666c6970fcdb071f01'
          /usr/bin/ld: $WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13.$CRATE.99a70f9b-cgu.0.rcgu.o: in function `<$CRATE::flipper::_::__ink_CallForwarder<&mut $CRATE::flipper::Flipper> as $CRATE::flipper::FlipperTrait>::new':
          $DIR/tests/ui/pass/08-flipper-as-dependency-trait.rs:3: undefined reference to `__ink_enforce_error_0x0230466c697070657254726169740c6e6577de27a252'
          /usr/bin/ld: $WORKSPACE/target/tests/target/x86_64-unknown-linux-gnu/debug/deps/$CRATE-52362a41e05cfb13.$CRATE.99a70f9b-cgu.0.rcgu.o: in function `<$CRATE::flipper::_::__ink_CallForwarder<&mut $CRATE::flipper::Flipper> as $CRATE::flipper::FlipperTrait>::get':
          $DIR/tests/ui/pass/08-flipper-as-dependency-trait.rs:3: undefined reference to `__ink_enforce_error_0x0130466c697070657254726169740c6765746110f2d800'
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

@TriplEight
Copy link
Contributor Author

dropping -Clink-dead-code works indeed

@TriplEight
Copy link
Contributor Author

TriplEight commented Jul 20, 2021

sadly codecov binary is of no use now, codecov/uploader#217 codecov/uploader#222. However, the old way with bash script works, I'll return to it.

@TriplEight TriplEight marked this pull request as ready for review July 20, 2021 16:05
Copy link
Collaborator

@cmichi cmichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM!

@TriplEight TriplEight changed the title Fix Codecov Publishing Fix Codecov measuring and Publishing Jul 20, 2021
Copy link
Collaborator

@Robbepop Robbepop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-Clink-dead-code is correct. The flipper example actually should just not generate the __ink_enforce_error in case it is a success test OR it should not be put under coverage if it is an enforced failure.

@TriplEight
Copy link
Contributor Author

TriplEight commented Jul 20, 2021

OK, now it works properly.
"Branches" coverage: https://codecov.io/github/Robbepop/ink/commit/fe84f31cfc839e7d1529977edbd854729a1f6f55
"Lines" coverage: https://codecov.io/github/paritytech/ink/commit/fe84f31cfc839e7d1529977edbd854729a1f6f55
Please notice that the values are similar, guess it's something to do with grcov again. I vaguely remember that source-based coverage can only be of one type.

.gitlab-ci.yml Outdated Show resolved Hide resolved
.gitlab-ci.yml Show resolved Hide resolved
.gitlab-ci.yml Show resolved Hide resolved
@TriplEight
Copy link
Contributor Author

Now, I'm not an expert in all those flags and in fact I wanted you guys to take a look at the quality of the coverage and come up with the decision whether it's good enough (better than previous).
And another point: I need to find a cite that source-based coverage does not support either lines or branches coverage. This will make the first upload to codecov obsolete.
With these two checks I will be ready to merge this PR

@TriplEight
Copy link
Contributor Author

TriplEight commented Jul 21, 2021

Did some research

See https://github.com/taiki-e/cargo-llvm-cov#continuous-integration and notes below

Currently, only line coverage is available on Codecov. This is because -Zinstrument-coverage does not support branch coverage and Codecov does not support region coverage.

Due to a bug of -Zinstrument-coverage, some files may be ignored. There is a known workaround for this issue, but note that the workaround is likely to cause another problem.

Also rust-lang/rust#86177

-Zinstrument-coverage seems to ignore files or modules where all items are dead code. The known workaround is using -Clink-dead-code, but it seems not the correct workaround and is likely to cause another problem.

Add support for branch coverage in source-based coverage #79649 rust-lang/rust#79649

Codecov doesn't support region coverage taiki-e/cargo-llvm-cov#12 taiki-e/cargo-llvm-cov#20

And a tracking issue about source-based coverage rust-lang/rust#79121

All of it means that the current state of the PR means that we are thrown back to line coverage.
Makes me think that it's better to use the old way with a pile of flags.
@Robbepop @cmichi ?

@HCastano
Copy link
Contributor

@Robbepop @cmichi pinging you guys for Denis' question

@TriplEight
Copy link
Contributor Author

Working MIR-based coverage lies in the commit above. However, it does not provide branches coverage and a lot of effort was put into it in ink!, so it was decided to return to the previous instrumentation.

@TriplEight TriplEight mentioned this pull request Aug 19, 2021
@cmichi
Copy link
Collaborator

cmichi commented Sep 13, 2021

@TriplEight Can we close this PR, since we've merged #895 now?

@TriplEight
Copy link
Contributor Author

yes, but I'll note it for myself as a working example of the source-based coverage.

@TriplEight TriplEight closed this Sep 13, 2021
@HCastano HCastano deleted the 3x8_fix_codecov_publishing branch July 29, 2022 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-CI Continuous integration work item
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants