[RNMobile] Initial HTML E2E test #26405
Conversation
jd-alexander
added
[Type] Automated Testing
jd-alexander
requested review from
karmatosed,
mapk and
youknowriad
as code owners
|
Size Change: 0 B Total Size: 1.42 MB ℹ️ View Unchanged
|
|
So as seen above the test was run successfully. I did download the video generated from Saucelabs and I noticed that the Android test video length was 3mins and the iOS test video length was 5 mins. Just noting that here for discussion :) |
|
Thanks a lot for this @jd-alexander 🎉
There could be some issues if you're using Xcode 12: #26283 (review). You can install Xcode 11 aside 12 for now until that PR is merged.
I think
I wasn't able to find the Sauce Labs video link for this test in the CircleCI output but I was able to watch it from GitHub Actions artifacts. For iOS I checked this run: https://github.com/WordPress/gutenberg/pull/26405/checks?check_run_id=1294846308. The most time consuming thing was typing the content in the text view. It took 5 minutes to type the whole initialHtml. I think a way to paste the initialHtml instead of typing on iOS would be really nice. It also didn't seem to scroll the list at all at the end of the test (maybe just a few pixels). For Android I checked the artifacts in this run: https://github.com/WordPress/gutenberg/pull/26405/checks?check_run_id=1294846311. It was very fast to type the content (maybe it was pasting) and the rest of the time (~2:30mins) was taken by scrolling. It was waiting 3 seconds each time like you mentioned but the scrolling itself seemed very slow as well. |
Thanks for your review @ceyhun 🙇
Thanks I am going to do that.
Nice 😄
Indeed! I am going to work on a way of doing the typing more instantaneously and yes, I am also going to look into why it doesn't do the scroll completely. I am wondering if it because it discover the element so it just stops. I will see what I can come up with.
Indeed. I am going to modify the function that does the swipes so that I can pass in a delay param so that the scrolling doesn't operate so slowly. |
|
So today I did some work to make these changes more optimized.
From an iOS standpoint, I still didn't get the test running on iOS 😞 This is the output I am getting I also tried to utilize the copy and paste functionality, within the app using this function I wrote below. The weird thing that I think is happening is the content being escaped by the driver. The escaped content looks like I haven't figured out what exactly is causing this as yet. Also while I was watching the video, I think the code that presses the paste button may be outdated so I will need to run a check on the coordinate algorithm being used to determine where on the screen should be clicked. |
Is the folder Are you using Xcode 11 on the command-line? You can check via running |
|
Thanks much @ceyhun that fixed the XCode issue. I am now having another issue So it seems the GutenbergDemo release project is not being generated
Yes, you are indeed correct with this one. Following the testing steps in the README.md file I ran the appium doctor commands for ios and I got the output below, which means I need to fix some issues ( some of them seem unrelated) so I will do that to see if it will help. |
|
@ceyhun would you mind giving me some help on the iOS side of things? I lessened the swipe delay and other updates and that is working great but on iOS, I still haven't figured out how to get the pasting working right. I wanted to dig a bit deeper today but I am wrangling the release process for the first time so it ending up consuming most of my time. So yes, let me know if you are able to have a look as I want us to get a working solution merged in so we can prevent breakages like the recent Verse breakage that was being wrangled today. Thanks for your help. |
|
I think I found a way to paste HTML instead of typing. Let's see if this passes on CI and how long it takes. If it takes too long, we can check if we can still catch the crashes if we don't scroll at all. Maybe just rendering the block list is enough. |
|
Yay 🎉🎊🎆 All the 8 are green ✅ Let's run 8 more!!! BTW here's the command I use to push empty commits: for i in `seq 1 8`; do git commit --allow-empty -m "Run $i" && git push; done |
Awesome!!!! 🥳
I will keep the reverting of these commits in mind. I am hoping the v2.15.0 really stabilises things on Android. Let's see what these 8 new runs have in store.
Pretty neat. Beats doing it manually 😅 |
|
The results are in! We got 16 consecutive ✅ successful runs @ceyhun Things are looking good 😎🚀🚇🚢 |
|
@ceyhun I just saw the conversation about flaky tests here 👍🏾 👍🏾 👍🏾 👍🏾 I am quite confident we are in a good place from a stability standpoint. When we do merge these changes we can monitor for stability and if any hiccups arise we have several Android-related desired capabilities ( in particular timeout-related caps ) that we can leverage to resolve any flaky issues that may occur. |
|
@jd-alexander I've reverted the commits deleting other workflows and also updated the gb-mobile PR with the gutenberg's current ref. If that's also green, I think we can go forward with both 🎉 |
| @@ -11,7 +11,7 @@ jobs: | |||
| strategy: | |||
| matrix: | |||
| native-test-name: [ | |||
| gutenberg-editor-gallery | |||
| gutenberg-editor-blocks | |||
There was a problem hiding this comment.
What do you think about renaming this to gutenberg-editor-initial-html and also keeping gutenberg-editor-gallery, but maybe making them run one after another instead of running in parallel?
There was a problem hiding this comment.
I agree with the renaming. What is the rationale for keeping the gallery test ? I thought this test would cover all cases and I'm wondering if that test could be more flaky than this one. Also I'm wondering about the collective time of both them running one after the other as well. Let me know what you think.
There was a problem hiding this comment.
To be honest, I'm not interested in gallery block in this case 😄 But the gallery test checks some other aspects of the editor that are not covered by the initial HTML tests, mainly around the inserter menu, like opening the inserter menu, scrolling the inserter menu, and inserting a block.
The primary time-consuming task is building the js bundle and the native code (the ipa and the apk). The gallery test itself runs in ~5 seconds on iOS and ~50 seconds on Android. Android is slower due to iOS having a larger inserter, so there's no need to scroll the inserter to find the gallery block on iOS.
Gallery test has been around for quite some time now, and I think it should be stable enough. But I realized now that we had made some changes around scrolling in this PR, that could be affecting the gallery test's stability. So it could be better to leave the gallery test out for now and maybe add it back in, in a separate PR making sure again it's flawless 💯
There was a problem hiding this comment.
To be honest, I'm not interested in gallery block in this case 😄 But the gallery test checks some other aspects of the editor that are not covered by the initial HTML tests, mainly around the inserter menu, like opening the inserter menu, scrolling the inserter menu, and inserting a block.
I understand. This makes a lot of sense.
The primary time-consuming task is building the js bundle and the native code (the ipa and the apk). The gallery test itself runs in ~5 seconds on iOS and ~50 seconds on Android. Android is slower due to iOS having a larger inserter, so there's no need to scroll the inserter to find the gallery block on iOS
Gallery test has been around for quite some time now, and I think it should be stable enough. But I realized now that we had made some changes around scrolling in this PR, that could be affecting the gallery test's stability. So it could be better to leave the gallery test out for now and maybe add it back in, in a separate PR making sure again it's flawless 💯
Sounds like a good plan to me to re-add it after we have done some more tests! Thanks for expounding here @ceyhun
Agreed. Thanks @ceyhun 😄 Once we have resolved the comment, we are good to go. |
Fixes #26146
Gutenberg Mobile PR wordpress-mobile/gutenberg-mobile#2753
Description
This change introduces a React Native E2E test that renders the initial HTML content that's used during development.
How has this been tested?
I tested this on Android, by running
TEST_RN_PLATFORM=android npm run native device-tests:local gutenberg-editor-blocks.test. I will await the results of this PR for iOS because I have been having some issues when running iOS tests locally.Types of changes
gutenbergPR has been changed from theGallerytest to theinitial htmltest.swipefunctionality within the test helper waits 3 seconds each time a swipe is done. I am assuming this is done to ensure that all elements on the screen are loaded and such forth so I didn't modify it. I am sharing this here in case this needs to be optimized.initial htmlcontent was added to thetest-data.jsfile. I am not sure if there's a way to include theinitial htmlbeing used in the other part of the app. If there is, please let me know so I can reduce the duplicate code. I just utilized this approach since I saw that all the test data resided in the test directory and I wasn't able to import theinitialHtmlmodule.Checklist: