Trying to fix integration tests #4842

ariskotsomitopoulos · 2022-01-03T23:05:30Z

Fixing Integration test

The main problem is that GitHub actions is that it has limited resources and is not meant for that kind of jobs. In order to properly do a robust 100% solution we should welcome another CI/CD tool like Jenkins and trigger the build from here

For further improvements we should change the actual implementation of the integration tests along with the shared resources

Synapse server can now start, it was broken
Fix/Improve running them on GitHub
Mark the not working tests with @ignore so that they will not run neither local or on github actions
Advanced parsing the result to add a user friendly comment on the PR with the results
Increase adb ram & also tried caching the adb

On mac slave it works much better and consistent

As you can see here the tests can run and the results are published like below:

github-actions · 2022-01-03T23:07:13Z

Unit Test Results

  66 files ±0   66 suites ±0 54s ⏱️ +3s
135 tests ±0 135 ✔️ ±0 0 💤 ±0 0 ❌ ±0
418 runs ±0 418 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 8adeab0. ± Comparison against base commit 1490854.

♻️ This comment has been updated with latest results.

@ignore

…ery very limited for that) - Split them in msdk and app test along with multiple smaller steps. - Mark the not working tests with @ignore so that they will not run neither local or on github actions - Add user friendly comment on PR to view the results

…ption

github-actions · 2022-01-05T22:24:02Z

Matrix SDK

Integration Tests Results:

[org.matrix.android.sdk.session]
passed="21" failures="8" errors="0" skipped="2"
[org.matrix.android.sdk.account]
passed="1" failures="1" errors="0" skipped="0"
[org.matrix.android.sdk.internal]
passed="1" failures="1" errors="0" skipped="0"
[org.matrix.android.sdk.ordering]
passed="1" failures="1" errors="0" skipped="0"
[org.matrix.android.sdk.PermalinkParserTest]
passed="1" failures="1" errors="0" skipped="0"

Undo 1 core usage Undo force-avd-creation

bmarty · 2022-01-07T15:25:56Z

We will have to update the GitHub setting to make the actions required:

Integration Tests / Matrix SDK - Build Android Tests (pull_request) - it was a temporary GitHub action, which can be removed now
Integration Tests / Matrix SDK - Running Integration Tests (28) (pull_request) - now passing!
Integration Tests / App - Build Android Tests (pull_request) - not required anymore, maybe because it has been renamed? Anyway it was a temporary GitHub action, which can be removed now)

Also the action Compile Android tests seems stuck.

bmarty

Nice work, thanks a lot for working on that.
Some comment/questions.

bmarty · 2022-01-07T15:34:52Z

.github/workflows/integration_tests.yml

+jobs:
+  # Build Android Tests [Matrix SDK]
+  build-android-test-matrix-sdk:
+    name: Matrix SDK - Build Android Tests


Do we want to keep this action since it is now done when we run the integration test?
It has been added (see the comment here) to ensure that the tests are compiling by making this action "Required" on every PRs.
But now I think we can put the test run "Required".

Thinking more about it, since this action is more reliable than the running of integration test, maybe we can keep it.

Exactly thats why I kept it. While its not 100% reliable I think we should keep it

bmarty · 2022-01-07T15:39:49Z

dependencies.gradle

@@ -29,6 +29,7 @@ def vanniktechEmoji = "0.8.0"
 def mockk = "1.12.1"
 def espresso = "3.4.0"
 def androidxTest = "1.4.0"
+def androidxOrchestrator = "1.4.1"


Why not just update the value of androidxTest above to 1.4.1?

As you can see below only androidxOrchestrator is yet it 1.4.1

You are right, I read a bit fast the title at https://developer.android.com/jetpack/androidx/releases/test#core_141_2

Also confirmed that this should not be a problem to use various version by the sample at https://developer.android.com/jetpack/androidx/releases/test#declaring_dependencies

bmarty · 2022-01-07T15:41:45Z

matrix-sdk-android/src/androidTest/java/org/matrix/android/sdk/common/RetryTestRule.kt

+                        return
+                    } catch (t: Throwable) {
+                        caughtThrowable = t
+                        Log.e(TAG, description.displayName + ": run " + (i + 1) + " failed")


Maybe also log the exception t?

Also can we use Timber here? (Maybe not)

matrix-sdk-android/src/androidTest/java/org/matrix/android/sdk/common/TestConstants.kt

bmarty · 2022-01-07T15:45:12Z

...droid/src/androidTest/java/org/matrix/android/sdk/internal/crypto/gossiping/WithHeldTests.kt

@@ -39,12 +41,14 @@ import org.matrix.android.sdk.internal.crypto.model.event.WithHeldCode

 @RunWith(AndroidJUnit4::class)
 @FixMethodOrder(MethodSorters.JVM)
+@LargeTest


Where are effectively use this annotation LargeTest (and the other ones)? It's maybe not in this PR?

Yes, we are not currently using those annotations. The plan is to start using this annotation to our tests when everything is annotated. I have annotated some of the tests but not all of them

bmarty · 2022-01-07T15:46:41Z

...-android/src/androidTest/java/org/matrix/android/sdk/internal/crypto/verification/SASTest.kt

@@ -528,7 +547,6 @@ class SASTest : InstrumentedTest {
        val aliceDeviceInfoFromBobPOV: CryptoDeviceInfo? = bobSession.cryptoService().getDeviceInfo(aliceSession.myUserId, aliceSession.cryptoService().getMyDevice().deviceId)

        // latch wait a bit again
-        Thread.sleep(1000)


Please also remove the comment above

bmarty · 2022-01-07T15:47:32Z

...androidTest/java/org/matrix/android/sdk/session/room/timeline/TimelineWithManyMembersTest.kt

@@ -31,8 +33,13 @@ import org.matrix.android.sdk.common.CommonTestHelper
 import org.matrix.android.sdk.common.CryptoTestHelper
 import java.util.concurrent.CountDownLatch

+/** !! Not working with the new timeline
+ *  Disabling it until the fix is made


…github.com/vector-im/element-android into feature/aris/integration_tests_improvement

github-actions · 2022-01-10T09:28:58Z

Ktlint Results

👍 ✅ 👍

bmarty

Thanks for the update

ariskotsomitopoulos added 2 commits January 5, 2022 23:42

Increase adbtimeout to prevent ShellCommandUnresponsiveException exce…

6a24e02

…ption

ariskotsomitopoulos force-pushed the feature/aris/integration_tests_improvement branch from 9221904 to 6a24e02 Compare January 5, 2022 21:45

element-hq deleted a comment from github-actions bot Jan 5, 2022

ariskotsomitopoulos added 4 commits January 6, 2022 00:38

fix typo

52c8a13

fix typo

abf8932

Remove adb timeout

41bf1cc

Use 1 core for the emulator

34efd91

ariskotsomitopoulos marked this pull request as ready for review January 7, 2022 10:09

ariskotsomitopoulos mentioned this pull request Jan 7, 2022

Github Actions Limitations #4876

Open

ariskotsomitopoulos and others added 2 commits January 7, 2022 12:44

try with force-avd-creation = true

2c8015c

Update integration_tests.yml

e2616ba

Undo 1 core usage Undo force-avd-creation

bmarty reviewed Jan 7, 2022

View reviewed changes

ariskotsomitopoulos added 2 commits January 10, 2022 11:26

PR Remarks

0fb26da

Merge branch 'feature/aris/integration_tests_improvement' of https://…

1224144

…github.com/vector-im/element-android into feature/aris/integration_tests_improvement

- fix lint error

8adeab0

bmarty approved these changes Jan 12, 2022

View reviewed changes

bmarty merged commit f8afe04 into develop Jan 12, 2022

bmarty deleted the feature/aris/integration_tests_improvement branch January 12, 2022 16:11

ouchadam mentioned this pull request Jan 14, 2022

In development release notes category #4953

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trying to fix integration tests #4842

Trying to fix integration tests #4842

ariskotsomitopoulos commented Jan 3, 2022 •

edited

Loading

github-actions bot commented Jan 3, 2022 •

edited

Loading

github-actions bot commented Jan 5, 2022 •

edited

Loading

bmarty commented Jan 7, 2022 •

edited

Loading

bmarty left a comment

bmarty Jan 7, 2022

ariskotsomitopoulos Jan 7, 2022

bmarty Jan 7, 2022

ariskotsomitopoulos Jan 7, 2022

bmarty Jan 7, 2022

bmarty Jan 7, 2022

bmarty Jan 7, 2022

bmarty Jan 7, 2022

ariskotsomitopoulos Jan 10, 2022

bmarty Jan 7, 2022

bmarty Jan 7, 2022

github-actions bot commented Jan 10, 2022

bmarty left a comment

Trying to fix integration tests #4842

Trying to fix integration tests #4842

Conversation

ariskotsomitopoulos commented Jan 3, 2022 • edited Loading

Fixing Integration test

github-actions bot commented Jan 3, 2022 • edited Loading

Unit Test Results

github-actions bot commented Jan 5, 2022 • edited Loading

Matrix SDK

Integration Tests Results:

bmarty commented Jan 7, 2022 • edited Loading

bmarty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2022

Ktlint Results

bmarty left a comment

Choose a reason for hiding this comment

ariskotsomitopoulos commented Jan 3, 2022 •

edited

Loading

github-actions bot commented Jan 3, 2022 •

edited

Loading

github-actions bot commented Jan 5, 2022 •

edited

Loading

bmarty commented Jan 7, 2022 •

edited

Loading