Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

395 e2e test #404

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft

395 e2e test #404

wants to merge 22 commits into from

Conversation

stefan-niedermann
Copy link
Member

This PR adds an end to end test to the SSO lib to fix #395.

Must:

  • Execution of connectedDebugAndroidTest fails (E2ETest.java)

Nice 2 Have:

  • Enable matrix testing in e2e.yml
  • latest.apk should be cached when matrix testing is used (e2e.yml)
  • Replace hard coded IP address 172.17.0.1 of the Nextcloud docker instance
  • Reenable AVD caching for quicker test runs
  • Pass details as arguments to connectedDebugAndroidTest instead of hard coded properties (E2ETest.java)

The E2E test has two steps after setting up the test environment:

  1. Setup account in the Nextcloud files app
  2. Import this account in the Sample app and verify a successful network call

The test completely uses UIAutomator (recommended by developer.android.com). It does not use Espresso or any other frameworks. Currently the first step works well, the second has an issue confirming the permission dialog because it can't find the OK button.

The workflow currently seems to work because i am running ./gradlew connectedDebugAndroidTest || true to suppress error mails, but it actually fails 😉 :

11-05 10:30:47.408  2823  2838 I E2E     : Configure Nextcloud account
11-05 10:30:47.412  2823  2838 D E2E     : Launching com.nextcloud.android.beta
11-05 10:31:06.372  2823  2838 D E2E     : Login Button exists. Clicking on it…
11-05 10:31:07.832  2823  2838 D E2E     : Login Button clicked.
11-05 10:31:08.390  2823  2838 D E2E     : URL input exists.
11-05 10:31:08.391  2823  2838 D E2E     : Entering URL…
11-05 10:31:09.149  2823  2838 D E2E     : URL entered.
11-05 10:31:09.149  2823  2838 D E2E     : Pressing enter…
11-05 10:31:09.341  2823  2838 D E2E     : Enter pressed.
11-05 10:31:09.341  2823  2838 D E2E     : Waiting for WebView…
11-05 10:31:37.525  2823  2838 D E2E     : WebView exists.
11-05 10:31:37.526  2823  2838 D E2E     : Waiting for WebView Login Button…
11-05 10:31:37.591  2823  2838 D E2E     : WebView Login Button exists. Clicking on it…
11-05 10:31:39.260  2823  2838 D E2E     : Waiting for Username Input…
11-05 10:32:09.669  2823  2838 D E2E     : Username Input exists. Setting text…
11-05 10:32:09.788  2823  2838 D E2E     : Username has been set.
11-05 10:32:09.799  2823  2838 D E2E     : Waiting for Password Input…
11-05 10:32:10.283  2823  2838 D E2E     : Password Input exists. Setting text…
11-05 10:32:11.169  2823  2838 D E2E     : Waiting for WebView Submit Button…
11-05 10:32:11.720  2823  2838 D E2E     : WebView Submit Button exists. Clicking on it…
11-05 10:32:13.182  2823  2838 D E2E     : Waiting for WebView Grant Access Button…
11-05 10:32:14.047  2823  2838 D E2E     : WebView Grant Access Button exists. Clicking on it…
11-05 10:32:18.175  2823  2838 I E2E     : Import account into sample app
11-05 10:32:18.175  2823  2838 D E2E     : Launching com.nextcloud.android.sso.sample
11-05 10:34:00.074  2823  2838 D E2E     : Waiting for OK Button…
11-05 10:35:01.678  2823  2838 D E2E     : OK Button exists. Clicking on it…

com.nextcloud.android.sso.sample.E2ETest > test_01_importAccountIntoSampleApp[test(AVD) - 7.0] FAILED 
	androidx.test.uiautomator.UiObjectNotFoundException: UiSelector[CONTAINS_TEXT=OK]
	at androidx.test.uiautomator.UiObject.click(UiObject.java:416)

Any help for the remaining steps would be appreciated @David-Development @tobiasKaminsky

@David-Development
Copy link
Member

@stefan-niedermann I did some testing on my end as well.. It looks like the mDevice.waitForWindowUpdate(null, TIMEOUT); just before the OK check is causing that issue. If I replace it with a Thread.sleep(5000) it works just fine - however it shows me only accounts from my normal nextcloud app - not the beta app.. Need to investigate that a little bit further why it's not showing up on my emulator

stefan-niedermann added a commit that referenced this pull request Nov 6, 2021
@stefan-niedermann
#395 Replace waitForWindowUpdate with Thread.sleep()
9c78081 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
@stefan-niedermann
Copy link
Member Author

@David-Development did as you suggested, but the pipeline still fails at the very same moment... 🤔

@David-Development
Copy link
Member

@stefan-niedermann Does it also fail if you run it locally? Can you see what's happening?

@stefan-niedermann
Copy link
Member Author

Okay, i stumbled upon AccountImporter#checkAndroidAccountPermissions found this in a previous log:

2021-11-05T09:53:56 E com.nextcloud.android.sso.AccountImporter: Permission not granted yet!
2021-11-05T09:53:56 W System.err: com.nextcloud.android.sso.exceptions.AndroidGetAccountsPermissionNotGranted: In order to use the Nextcloud Single-Sign-On on this device, you need to grant permission to access your accounts
2021-11-05T09:53:57 W System.err: 	at com.nextcloud.android.sso.AccountImporter.checkAndroidAccountPermissions(AccountImporter.java:124)
2021-11-05T09:53:57 W System.err: 	at com.nextcloud.android.sso.AccountImporter.pickNewAccount(AccountImporter.java:86)
2021-11-05T09:53:57 W System.err: 	at com.nextcloud.android.sso.sample.MainActivity.lambda$onCreate$0$com-nextcloud-android-sso-sample-MainActivity(MainActivity.java:43)
2021-11-05T09:53:57 W System.err: 	at com.nextcloud.android.sso.sample.MainActivity$$ExternalSyntheticLambda0.onClick(Unknown Source)
2021-11-05T09:53:57 W System.err: 	at android.view.View.performClick(View.java:5610)
2021-11-05T09:53:57 W System.err: 	at com.google.android.material.button.MaterialButton.performClick(MaterialButton.java:1119)
2021-11-05T09:53:57 W System.err: 	at android.view.View$PerformClick.run(View.java:22265)
2021-11-05T09:53:57 W System.err: 	at android.os.Handler.handleCallback(Handler.java:751)
2021-11-05T09:53:57 W System.err: 	at android.os.Handler.dispatchMessage(Handler.java:95)
2021-11-05T09:53:57 W System.err: 	at android.os.Looper.loop(Looper.java:154)
2021-11-05T09:53:57 W System.err: 	at android.app.ActivityThread.main(ActivityThread.java:6077)
2021-11-05T09:53:57 W System.err: 	at java.lang.reflect.Method.invoke(Native Method)
2021-11-05T09:53:57 W System.err: 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
2021-11-05T09:53:57 W System.err: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)
2021-11-05T09:53:57 W audio_hw_generic: Not supplying enough data to HAL, expected position 155345 , only wrote 154800

Therefore i now adjusted the AVD emulated device to Android O (API 26) to check whether this might be the cause of the issue - Maybe on older devices the OK button is actually not present because a permission dialog is there. Let's see how it goes...

@stefan-niedermann
Copy link
Member Author

Just fyi: This PR is waiting for a new dev version being published on https://download.nextcloud.com/android/dev/ - the currently latest version (nextcloud-dev-20211111.apk) does not contain the QueryParam fix which caused SSO 0.6.0 to not being able to connect to this files app version.

Since 3.18.0 RC6 has already been tagged, i think it's just a matter of time until we can continue here 🙂

@nextcloud-android-bot
Copy link
Collaborator

Codacy

Lint

TypemasterPR
Warnings11
Errors00

SpotBugs (new)

Warning Type Number
Bad practice Warnings 3
Correctness Warnings 21
Internationalization Warnings 2
Multithreaded correctness Warnings 3
Performance Warnings 5
Security Warnings 5
Dodgy code Warnings 43
Total 82

SpotBugs (master)

Warning Type Number
Bad practice Warnings 3
Correctness Warnings 21
Internationalization Warnings 2
Multithreaded correctness Warnings 3
Performance Warnings 5
Security Warnings 5
Dodgy code Warnings 43
Total 82

AndyScherzinger pushed a commit that referenced this pull request Jan 19, 2024
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
343c6d2 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
@AndyScherzinger AndyScherzinger force-pushed the 395-e2e-test branch 2 times, most recently from e146314 to 8bfcd55 Compare January 19, 2024 16:56
AndyScherzinger pushed a commit that referenced this pull request Jan 19, 2024
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
dfe0909 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
AndyScherzinger pushed a commit that referenced this pull request Jan 19, 2024
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
fdc4668 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
AndyScherzinger pushed a commit that referenced this pull request Jan 19, 2024
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
0468046 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@AndyScherzinger
Copy link
Member

❗ REBASED ❗

AndyScherzinger pushed a commit that referenced this pull request Jan 19, 2024
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
2d3410e 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
AndyScherzinger pushed a commit that referenced this pull request Feb 16, 2024
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
d6e352a 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
stefan-niedermann and others added 8 commits March 4, 2024 20:23
@stefan-niedermann @AndyScherzinger
#395 Add E2E test
df4e839 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
#395 E2E test
9053036 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
#395 Disable caching for AVD
9286b85 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Limit logs to com.nextcloud.* packages
d537150 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Enhance logging
ec1c8da 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Merge import and verification tests
ecf97ea 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Limit logs to "E2E" tag
597d454 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Configure trusted domain in env variable
20883cd 
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
#395 Replace waitForWindowUpdate with Thread.sleep()
c2366ab 
As suggested in #404 (comment)

Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Add quite a lot of Thread.sleep()s...
5a6eaa9 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Use AVD API 26
e23f8f6 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
More logs
d57337b 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Parallel SDK 24 & 26 without failfast
e505cc9 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Readd logs from e2e test
75b0c65 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Unify waits in e2e test
f160a52 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
Use Android 28 for emulated device
c5fbcae 
ReactiveCircus/android-emulator-runner#49 (comment)
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
chore(e2e): Move setup from CI to gradle task for easier local run
4dd6784 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
chore(e2e): Fail e2e.yml when e2e test fails
2de4814 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
chore(e2e): Add more log output
12175ed 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
chore(e2e): Run e2e test only for sample module
3043c62 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
chore(e2e): Move emulator setup back to CI
339cbaa 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
@stefan-niedermann @AndyScherzinger
chore(e2e): Enhance documentation
eac73ab 
Signed-off-by: Stefan Niedermann <info@niedermann.it>
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
2. developing Work in progress help wanted
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Single Sign On E2E test concept
4 participants
@stefan-niedermann @David-Development @nextcloud-android-bot @AndyScherzinger