ci: run tests on Android emulator

[mxinden]

Pendant to quinn-rs/quinn#1950.

Fixes #2028.

[mxinden]

Failing with:

android-actions/setup-android@v2 and reactivecircus/android-emulator-runner@v2 are not allowed to be used in mozilla/neqo.

@larseggert where do I find Mozilla's list of allowed GitHub actions?

[larseggert]

https://github.com/mozilla/neqo/settings/actions under "General" has the list. Let me know if that is not visible to you.

[github-actions]

Failed Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

• neqo-latest vs. aioquic: Z
• neqo-latest vs. haproxy: L1 C1
• neqo-latest vs. lsquic: C1
• neqo-latest vs. msquic: A C1
• neqo-latest vs. mvfst: A L1 C1
• neqo-latest vs. quiche: L1 C1
• neqo-latest vs. quinn: L1
• neqo-latest vs. xquic: A

neqo-latest as server

• lsquic vs. neqo-latest: run cancelled after 20 min
• msquic vs. neqo-latest: Z
• mvfst vs. neqo-latest: Z A L1 C1
• quinn vs. neqo-latest: V2
• xquic vs. neqo-latest: M

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

• neqo-latest vs. aioquic: H DC LR C20 M S R 3 B U A L1 L2 C1 C2 6 V2
• neqo-latest vs. go-x-net: H DC LR M B U A L2 C2 6
• neqo-latest vs. haproxy: H DC LR C20 M S R Z 3 B U A L2 C2 6 V2
• neqo-latest vs. kwik: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
• neqo-latest vs. lsquic: H DC LR C20 M S R Z 3 B U E A L1 L2 C2 6 V2
• neqo-latest vs. msquic: H DC LR C20 M S R Z B U L1 L2 C2 6 V2
• neqo-latest vs. mvfst: H DC LR M R Z 3 B U L2 C2 6
• neqo-latest vs. neqo: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• neqo-latest vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• neqo-latest vs. nginx: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
• neqo-latest vs. ngtcp2: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• neqo-latest vs. picoquic: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• neqo-latest vs. quic-go: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
• neqo-latest vs. quiche: H DC LR C20 M S R Z 3 B U A L2 C2 6
• neqo-latest vs. quinn: H DC LR C20 M S R Z 3 B U E A L2 C1 C2 6
• neqo-latest vs. s2n-quic: H DC LR C20 M S R 3 B U E A L1 L2 C1 C2 6
• neqo-latest vs. xquic: H DC LR C20 M R Z 3 B U L1 L2 C1 C2 6

neqo-latest as server

• aioquic vs. neqo-latest: H DC LR C20 M S R Z 3 B A L1 L2 C1 C2 6 V2
• chrome vs. neqo-latest: 3
• go-x-net vs. neqo-latest: H DC LR M B U A L2 C2 6
• kwik vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
• msquic vs. neqo-latest: H DC LR C20 M S R B U A L1 L2 C1 C2 6 V2
• mvfst vs. neqo-latest: H DC LR M 3 B L2 C2 6
• neqo vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• ngtcp2 vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• picoquic vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• quic-go vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
• quiche vs. neqo-latest: H DC LR M S R Z 3 B A L1 L2 C1 C2 6
• quinn vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6
• s2n-quic vs. neqo-latest: H DC LR M S R 3 B E A L1 L2 C1 C2 6
• xquic vs. neqo-latest: H DC LR C20 S R Z 3 B U A L1 L2 C1 C2 6

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

• neqo-latest vs. aioquic: E
• neqo-latest vs. go-x-net: C20 S R Z 3 E L1 C1 V2
• neqo-latest vs. haproxy: E
• neqo-latest vs. kwik: E
• neqo-latest vs. msquic: 3 E
• neqo-latest vs. mvfst: C20 S E V2
• neqo-latest vs. nginx: E V2
• neqo-latest vs. quic-go: E V2
• neqo-latest vs. quiche: E V2
• neqo-latest vs. quinn: V2
• neqo-latest vs. s2n-quic: Z V2
• neqo-latest vs. xquic: S E V2

neqo-latest as server

• aioquic vs. neqo-latest: U E
• chrome vs. neqo-latest: H DC LR C20 M S R Z B U E A L1 L2 C1 C2 6 V2
• go-x-net vs. neqo-latest: C20 S R Z 3 E L1 C1 V2
• kwik vs. neqo-latest: E
• msquic vs. neqo-latest: 3 E
• mvfst vs. neqo-latest: C20 S R U E V2
• quic-go vs. neqo-latest: E V2
• quiche vs. neqo-latest: C20 U E V2
• s2n-quic vs. neqo-latest: C20 Z U V2
• xquic vs. neqo-latest: E V2

[github-actions]

Firefox builds for this PR

The following builds are available for testing. Crossed-out builds did not succeed.

• Linux: Debug Release
• macOS: Debug Release
• Windows: Debug Release

[larseggert]

@mxinden did you ask for the actions to be allowlisted? Should I?

[mxinden]

I have not gotten to it, sorry. Help appreciated. I assume reactivecircus/android-emulator-runner is the most important one, i.e. most difficult to write ourselves.

[larseggert]

@mxinden these actions should now be available: https://bugzilla.mozilla.org/show_bug.cgi?id=1931529

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.37%. Comparing base (2fe2922) to head (6f9b8d9).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2058   +/-   ##
=======================================
  Coverage   95.37%   95.37%           
=======================================
  Files         112      112           
  Lines       36569    36569           
=======================================
  Hits        34879    34879           
  Misses       1690     1690           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[mxinden]

Cross-compilation of NSS for Android on Linux is failing. I will look into this.

[larseggert]

It's probably because it is grabbing the precompiled Linux build for NSS from the action cache. We need to differentiate between Linux and Android when building/caching NSS.

[larseggert]

We need to discuss with the NSS team how they build for Android. I have the suspicion that there are again no instructions, and just the way that the Firefox build does it is working. Topic for Thursday.