Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RNMobile] Initial HTML E2E test #26405

Merged
merged 85 commits into from
Mar 31, 2021
Merged
Show file tree
Hide file tree
Changes from 83 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
34ec8f0
added initial-html to test-data.js
jd-alexander Oct 22, 2020
e7194c6
Added a test that loads all the blocks within a post.
jd-alexander Oct 22, 2020
e94dce5
Updated the ci test runners to run gutenberg-editor-blocks
jd-alexander Oct 22, 2020
4f76cbc
added a function that scrolls to a specific element and returns it.
jd-alexander Oct 22, 2020
816b273
updated the test to utilize the scroll and return functionality
jd-alexander Oct 22, 2020
bdd999b
utilize a shared initialHtml for both test and initial editor content.
jd-alexander Oct 26, 2020
15ff496
added delay to swipe methods.
jd-alexander Oct 26, 2020
aaf6f9a
Merge branch 'master' into rnmobile/initial_html_test
ceyhun Jan 15, 2021
fb5b445
Remove unneeded code after merging master
ceyhun Jan 15, 2021
5a4a97f
Paste instead of typing when setting HTML content
ceyhun Jan 15, 2021
ae7eee6
Paste using keycode on Android
ceyhun Jan 18, 2021
c6c8f83
Select html content view only on android
ceyhun Jan 18, 2021
85178d1
Use iOS 14.0 simulator on server
ceyhun Jan 19, 2021
f136234
Print driver data on stop
ceyhun Jan 19, 2021
6749d83
Enable remaining paragraph tests on iOS
ceyhun Jan 19, 2021
99d028d
Remove dot at the end of jobURL
ceyhun Jan 19, 2021
9e16a13
Print menuButton
ceyhun Jan 19, 2021
cf9f538
Print error if environment cannot be initialized
ceyhun Jan 19, 2021
4f9e027
Revert "Use iOS 14.0 simulator on server"
ceyhun Jan 20, 2021
b8bce5c
Revert "Print driver data on stop"
ceyhun Jan 20, 2021
ffa7871
Revert "Print menuButton"
ceyhun Jan 20, 2021
a26ae92
Update server appium version to 1.18 and iOS to 14.0
ceyhun Jan 20, 2021
9401e28
Add teardown to paste test
ceyhun Jan 20, 2021
e8402b3
On iOS wait for paste notification to disappear
ceyhun Jan 21, 2021
874319e
Double tap instead of long press to paste
ceyhun Jan 21, 2021
63fa606
Fix lint error
ceyhun Jan 21, 2021
7a03e76
On iOS also long press before pasting
ceyhun Jan 21, 2021
206048e
On iOS use click instead of clickBeginningOfElement in paragraph test
ceyhun Jan 21, 2021
fabb5b4
Scroll to bottom by adding a paragraph block to the end
ceyhun Jan 21, 2021
20ab417
Use .type on Android instead of pressing paste keycode
ceyhun Jan 22, 2021
17c0ac9
Merge branch 'master' into rnmobile/initial_html_test
ceyhun Feb 3, 2021
cde7701
Check last block without scrolling on iOS
ceyhun Feb 4, 2021
6b50987
Update package-lock.json
ceyhun Feb 4, 2021
59f3b94
added endYCoefficient to increase the swipe distance on Android.
jd-alexander Feb 16, 2021
26c8f76
Merge branch 'master' into rnmobile/initial_html_test
jd-alexander Feb 16, 2021
2c84369
added package-lock.json
jd-alexander Feb 16, 2021
8954975
Merge branch 'trunk' into rnmobile/initial_html_test
jd-alexander Mar 2, 2021
5d1a932
Merge branch 'trunk' into rnmobile/initial_html_test
jd-alexander Mar 3, 2021
1a52d15
Merge branch 'trunk' into rnmobile/initial_html_test
ceyhun Mar 17, 2021
552721a
Merge branch 'trunk' into rnmobile/initial_html_test
jd-alexander Mar 20, 2021
be5d71b
Retry one more time getting the last block after a delay on iOS
ceyhun Mar 23, 2021
fe5a1b1
Temporarily delete cancel workflow (and others) to run native jobs mu…
ceyhun Mar 23, 2021
95b94ef
Run 1
ceyhun Mar 23, 2021
4e82557
Run 2
ceyhun Mar 23, 2021
36436c6
Run 3
ceyhun Mar 23, 2021
fc56a71
Run 4
ceyhun Mar 23, 2021
847114d
Run 5
ceyhun Mar 23, 2021
ace54df
Run 6
ceyhun Mar 23, 2021
4237c56
Run 7
ceyhun Mar 23, 2021
caf324c
Run 8
ceyhun Mar 23, 2021
6433d2a
Revert "Temporarily delete cancel workflow (and others) to run native…
ceyhun Mar 23, 2021
85dccc7
Merge branch 'trunk' into rnmobile/initial_html_test
ceyhun Mar 23, 2021
b31695a
Temporarily delete cancel workflow
ceyhun Mar 23, 2021
3eef3c6
Run 1
ceyhun Mar 23, 2021
99a2457
Run 2
ceyhun Mar 23, 2021
d369139
Run 3
ceyhun Mar 23, 2021
5346120
Run 4
ceyhun Mar 23, 2021
9f7c575
Run 5
ceyhun Mar 23, 2021
8c7d926
Run 6
ceyhun Mar 23, 2021
80f79c5
Run 7
ceyhun Mar 23, 2021
a49bd8a
Run 8
ceyhun Mar 23, 2021
0ec4ca5
Merge branch 'trunk' into rnmobile/initial_html_test
ceyhun Mar 25, 2021
ff5a3c4
Temporarily delete non-native workflows
ceyhun Mar 25, 2021
8944637
Bump reactivecircus/android-emulator-runner action to v2.15.0
ceyhun Mar 25, 2021
e95740f
Run 1
ceyhun Mar 25, 2021
7185703
Run 2
ceyhun Mar 25, 2021
14aff0f
Run 3
ceyhun Mar 25, 2021
a281149
Run 4
ceyhun Mar 25, 2021
968ec08
Run 5
ceyhun Mar 25, 2021
50795ad
Run 6
ceyhun Mar 25, 2021
7996123
Run 7
ceyhun Mar 25, 2021
7c656e2
Run 8
ceyhun Mar 25, 2021
fc6353f
Run 9
ceyhun Mar 25, 2021
b916bd4
Run 10
ceyhun Mar 25, 2021
9e0db5b
Run 11
ceyhun Mar 25, 2021
667c99a
Run 12
ceyhun Mar 25, 2021
972a472
Run 13
ceyhun Mar 25, 2021
603f27e
Run 14
ceyhun Mar 25, 2021
78bfdf0
Run 15
ceyhun Mar 25, 2021
376373a
Run 16
ceyhun Mar 25, 2021
3de14a6
Merge branch 'trunk' into rnmobile/initial_html_test
ceyhun Mar 30, 2021
ad6e0d7
Revert "Temporarily delete cancel workflow"
ceyhun Mar 30, 2021
d5a33b5
Revert "Temporarily delete non-native workflows"
ceyhun Mar 30, 2021
b3a63cb
Merge branch 'trunk' into rnmobile/initial_html_test
ceyhun Mar 31, 2021
6ad45cc
Rename test file
ceyhun Mar 31, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/rnmobile-android-runner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
matrix:
native-test-name: [
gutenberg-editor-gallery
gutenberg-editor-blocks
Copy link
Member

@ceyhun ceyhun Mar 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about renaming this to gutenberg-editor-initial-html and also keeping gutenberg-editor-gallery, but maybe making them run one after another instead of running in parallel?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the renaming. What is the rationale for keeping the gallery test ? I thought this test would cover all cases and I'm wondering if that test could be more flaky than this one. Also I'm wondering about the collective time of both them running one after the other as well. Let me know what you think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I'm not interested in gallery block in this case 😄 But the gallery test checks some other aspects of the editor that are not covered by the initial HTML tests, mainly around the inserter menu, like opening the inserter menu, scrolling the inserter menu, and inserting a block.

The primary time-consuming task is building the js bundle and the native code (the ipa and the apk). The gallery test itself runs in ~5 seconds on iOS and ~50 seconds on Android. Android is slower due to iOS having a larger inserter, so there's no need to scroll the inserter to find the gallery block on iOS.

Gallery test has been around for quite some time now, and I think it should be stable enough. But I realized now that we had made some changes around scrolling in this PR, that could be affecting the gallery test's stability. So it could be better to leave the gallery test out for now and maybe add it back in, in a separate PR making sure again it's flawless 💯

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I'm not interested in gallery block in this case 😄 But the gallery test checks some other aspects of the editor that are not covered by the initial HTML tests, mainly around the inserter menu, like opening the inserter menu, scrolling the inserter menu, and inserting a block.

I understand. This makes a lot of sense.

The primary time-consuming task is building the js bundle and the native code (the ipa and the apk). The gallery test itself runs in ~5 seconds on iOS and ~50 seconds on Android. Android is slower due to iOS having a larger inserter, so there's no need to scroll the inserter to find the gallery block on iOS
Gallery test has been around for quite some time now, and I think it should be stable enough. But I realized now that we had made some changes around scrolling in this PR, that could be affecting the gallery test's stability. So it could be better to leave the gallery test out for now and maybe add it back in, in a separate PR making sure again it's flawless 💯

Sounds like a good plan to me to re-add it after we have done some more tests! Thanks for expounding here @ceyhun

]

steps:
Expand All @@ -35,7 +35,7 @@ jobs:
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle

- uses: reactivecircus/android-emulator-runner@08b092e904025fada32a01b711af1e7ff7b7a4a3 # v2.14.3
- uses: reactivecircus/android-emulator-runner@d2799957d660add41c61a5103e2fbb9e2889eb73 # v2.15.0
with:
api-level: 28
profile: pixel_xl
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/rnmobile-ios-runner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
matrix:
xcode: [12.2]
native-test-name: [
gutenberg-editor-gallery
gutenberg-editor-blocks
]

steps:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/**
* Internal dependencies
*/
import initialHtml from '../src/initial-html';
import { isAndroid } from './helpers/utils';

describe( 'Gutenberg Editor Blocks test', () => {
it( 'should be able to create a post with all blocks and scroll to the last one', async () => {
await editorPage.setHtmlContent( initialHtml );

const lastBlockAccessibilityLabel =
'This block is used in initial HTML e2e tests and should be kept as the last block.';
let lastBlockElement;
if ( isAndroid() ) {
lastBlockElement = await editorPage.androidScrollAndReturnElement(
lastBlockAccessibilityLabel
);
} else {
lastBlockElement = await editorPage.getLastElementByXPath(
lastBlockAccessibilityLabel
);
if ( ! lastBlockElement ) {
const retryDelay = 5000;
// eslint-disable-next-line no-console
console.log(
`Warning: "lastBlockElement" was not found in the first attempt. Could be that all the blocks were not loaded yet.
Will retry one more time after ${ retryDelay / 1000 } seconds.`,
lastBlockElement
);
await editorPage.driver.sleep( retryDelay );
lastBlockElement = await editorPage.getLastElementByXPath(
lastBlockAccessibilityLabel
);
}
}

expect( lastBlockElement ).toBeTruthy();
} );
} );
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,8 @@ describe( 'Gutenberg Editor tests for Paragraph Block', () => {
}
} );

// Restricting these test to Android because I was not able to update the html on iOS
if ( isAndroid() ) {
it( 'should be able to merge blocks with unknown html elements', async () => {
await editorPage.setHtmlContent( `
it( 'should be able to merge blocks with unknown html elements', async () => {
await editorPage.setHtmlContent( `
<!-- wp:paragraph -->
<p><unknownhtmlelement>abc</unknownhtmlelement>D</p>
<!-- /wp:paragraph -->
Expand All @@ -160,61 +158,53 @@ describe( 'Gutenberg Editor tests for Paragraph Block', () => {
<p>E</p>
<!-- /wp:paragraph -->` );

// // Merge paragraphs
const secondParagraphBlockElement = await editorPage.getBlockAtPosition(
blockNames.paragraph,
2
);
await clickBeginningOfElement(
editorPage.driver,
secondParagraphBlockElement
);
await editorPage.typeTextToParagraphBlock(
secondParagraphBlockElement,
backspace
);

// verify the editor has not crashed
const text = await editorPage.getTextForParagraphBlockAtPosition(
1
);
expect( text.length ).not.toEqual( 0 );

await editorPage.removeBlockAtPosition( blockNames.paragraph );
} );

// Based on https://github.com/wordpress-mobile/gutenberg-mobile/pull/1507
it( 'should handle multiline paragraphs from web', async () => {
await editorPage.setHtmlContent( `
<!-- wp:paragraph -->
<p>multiple lines<br><br></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->` );

// // Merge paragraphs
const secondParagraphBlockElement = await editorPage.getBlockAtPosition(
blockNames.paragraph,
2
);
await clickBeginningOfElement(
editorPage.driver,
secondParagraphBlockElement
);
await editorPage.typeTextToParagraphBlock(
secondParagraphBlockElement,
backspace
);

// verify the editor has not crashed
const text = await editorPage.getTextForParagraphBlockAtPosition(
1
);
expect( text.length ).not.toEqual( 0 );

await editorPage.removeBlockAtPosition( blockNames.paragraph );
} );
}
// // Merge paragraphs
const secondParagraphBlockElement = await editorPage.getBlockAtPosition(
blockNames.paragraph,
2
);
await clickBeginningOfElement(
editorPage.driver,
secondParagraphBlockElement
);
await editorPage.typeTextToParagraphBlock(
secondParagraphBlockElement,
backspace
);

// verify the editor has not crashed
const text = await editorPage.getTextForParagraphBlockAtPosition( 1 );
expect( text.length ).not.toEqual( 0 );

await editorPage.removeBlockAtPosition( blockNames.paragraph );
} );

// Based on https://github.com/wordpress-mobile/gutenberg-mobile/pull/1507
it( 'should handle multiline paragraphs from web', async () => {
await editorPage.setHtmlContent( `
<!-- wp:paragraph -->
<p>multiple lines<br><br></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->` );

// // Merge paragraphs
const secondParagraphBlockElement = await editorPage.getBlockAtPosition(
blockNames.paragraph,
2
);
await secondParagraphBlockElement.click();
await editorPage.typeTextToParagraphBlock(
secondParagraphBlockElement,
backspace
);

// verify the editor has not crashed
const text = await editorPage.getTextForParagraphBlockAtPosition( 1 );
expect( text.length ).not.toEqual( 0 );

await editorPage.removeBlockAtPosition( blockNames.paragraph );
} );
} );
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ describe( 'Gutenberg Editor paste tests', () => {

const text = await editorPage.getTextForParagraphBlockAtPosition( 2 );
expect( text ).toBe( testData.pastePlainText );

await editorPage.removeBlockAtPosition( blockNames.paragraph, 2 );
await editorPage.removeBlockAtPosition( blockNames.paragraph, 1 );
} );

it( 'copies styled text from one paragraph block and pastes in another', async () => {
Expand Down
6 changes: 3 additions & 3 deletions packages/react-native-editor/__device-tests__/helpers/caps.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ const ios = {
os: 'iOS',
deviceOrientation: 'portrait',
automationName: 'XCUITest',
appiumVersion: '1.17.1', // Sauce Labs requires appiumVersion to be specified.
appiumVersion: '1.18.3', // Sauce Labs requires appiumVersion to be specified.
app: undefined, // will be set later, locally this is relative to root of project
processArguments: {
args: [ 'uitesting' ],
Expand All @@ -20,7 +20,7 @@ exports.iosLocal = {

exports.iosServer = {
...ios,
platformVersion: '13.4', // Supported Sauce Labs platforms can be found here: https://saucelabs.com/rest/v1/info/platforms/appium
platformVersion: '14.0', // Supported Sauce Labs platforms can be found here: https://saucelabs.com/rest/v1/info/platforms/appium
deviceName: 'iPhone 11 Simulator',
};

Expand All @@ -34,6 +34,6 @@ exports.android = {
appPackage: 'com.gutenberg',
appActivity: 'com.gutenberg.MainActivity',
deviceOrientation: 'portrait',
appiumVersion: '1.16.0',
appiumVersion: '1.18.1',
app: undefined,
};
43 changes: 32 additions & 11 deletions packages/react-native-editor/__device-tests__/helpers/utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ const stopDriver = async ( driver ) => {
.createHmac( 'md5', jobID )
.update( serverConfigs.sauce.auth )
.digest( 'hex' );
const jobURL = `https://saucelabs.com/jobs/${ jobID }?auth=${ hash }.`;
const jobURL = `https://saucelabs.com/jobs/${ jobID }?auth=${ hash }`;
// eslint-disable-next-line no-console
console.log( `You can view the video of this test run at ${ jobURL }` );
}
Expand Down Expand Up @@ -218,15 +218,19 @@ const clearTextBox = async ( driver, element ) => {
// We are double tapping on the text field and pressing backspace until all content is removed.
do {
originalText = await element.text();
const action = new wd.TouchAction( driver );
action.tap( { el: element, count: 2 } );
await action.perform();
await doubleTap( driver, element );
await element.type( '\b' );
text = await element.text();
// We compare with the original content and not empty because text always return any hint set on the element.
} while ( originalText !== text );
};

const doubleTap = async ( driver, element ) => {
const action = new wd.TouchAction( driver );
action.tap( { el: element, count: 2 } );
await action.perform();
};

const typeStringAndroid = async (
driver,
element,
Expand Down Expand Up @@ -351,7 +355,12 @@ const tapPasteAboveElement = async ( driver, element ) => {

// Starts from the middle of the screen or the element(if specified)
// and swipes upwards
const swipeUp = async ( driver, element = undefined ) => {
const swipeUp = async (
driver,
element = undefined,
delay = 3000,
endYCoefficient = 0.5
) => {
let size = await driver.getWindowSize();
let y = 0;
if ( element !== undefined ) {
Expand All @@ -363,27 +372,33 @@ const swipeUp = async ( driver, element = undefined ) => {
const startX = size.width / 2;
const startY = y + size.height / 3;
const endX = startX;
const endY = startY + startY * -1 * 0.5;
const endY = startY + startY * -1 * endYCoefficient;

await swipeFromTo( driver, { x: startX, y: startY }, { x: endX, y: endY } );
await swipeFromTo(
driver,
{ x: startX, y: startY },
{ x: endX, y: endY },
delay
);
};

const defaultCoordinates = { x: 0, y: 0 };
const swipeFromTo = async (
driver,
from = defaultCoordinates,
to = defaultCoordinates
to = defaultCoordinates,
delay
) => {
const action = await new wd.TouchAction( driver );
action.press( from );
action.wait( 3000 );
action.wait( delay );
action.moveTo( to );
action.release();
await action.perform();
};

// Starts from the middle of the screen and swipes downwards
const swipeDown = async ( driver ) => {
const swipeDown = async ( driver, delay = 3000 ) => {
const size = await driver.getWindowSize();
const y = 0;

Expand All @@ -392,7 +407,12 @@ const swipeDown = async ( driver ) => {
const endX = startX;
const endY = startY - startY * -1 * 0.5;

await swipeFromTo( driver, { x: startX, y: startY }, { x: endX, y: endY } );
await swipeFromTo(
driver,
{ x: startX, y: startY },
{ x: endX, y: endY },
delay
);
};

const toggleHtmlMode = async ( driver, toggleOn ) => {
Expand Down Expand Up @@ -451,4 +471,5 @@ module.exports = {
stopDriver,
toggleHtmlMode,
toggleOrientation,
doubleTap,
};
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const {
toggleHtmlMode,
swipeFromTo,
longPressMiddleOfElement,
doubleTap,
} = require( '../helpers/utils' );

const initializeEditorPage = async () => {
Expand Down Expand Up @@ -135,6 +136,27 @@ class EditorPage {
return elements[ elements.length - 1 ];
}

// iOS loads the block list more eagerly compared to Android.
// This makes this function return elements without scrolling on iOS.
// So we are keeping this Android only.
async androidScrollAndReturnElement( accessibilityLabel ) {
const elements = await this.driver.elementsByXPath(
`//*[contains(@${ this.accessibilityIdXPathAttrib }, "${ accessibilityLabel }")]`
);
if ( elements.length === 0 ) {
await swipeUp( this.driver, undefined, 100, 1 );
return this.androidScrollAndReturnElement( accessibilityLabel );
}
return elements[ elements.length - 1 ];
}

async getLastElementByXPath( accessibilityLabel ) {
const elements = await this.driver.elementsByXPath(
`//*[contains(@${ this.accessibilityIdXPathAttrib }, "${ accessibilityLabel }")]`
);
return elements[ elements.length - 1 ];
}

async getTextViewForHtmlViewContent() {
const accessibilityId = 'html-view-content';
let blockLocator = `//*[@${ this.accessibilityIdXPathAttrib }="${ accessibilityId }"]`;
Expand All @@ -161,8 +183,30 @@ class EditorPage {
async setHtmlContent( html ) {
await toggleHtmlMode( this.driver, true );

const base64String = Buffer.from( html ).toString( 'base64' );

await this.driver.setClipboard( base64String, 'plaintext' );

const htmlContentView = await this.getTextViewForHtmlViewContent();
await htmlContentView.type( html );

if ( isAndroid() ) {
// Attention! On Android `.type()` replaces the content of htmlContentView instead of appending
// contrary to what iOS is doing. On Android tried calling `driver.pressKeycode( 279 ) // KEYCODE_PASTE`
// before to paste, but for some reason it didn't work on GitHub Actions but worked only on Sauce Labs
await htmlContentView.type( html );
} else {
await htmlContentView.click();
await doubleTap( this.driver, htmlContentView );
// Sometimes double tap is not enough for paste menu to appear, so we also long press
await longPressMiddleOfElement( this.driver, htmlContentView );

const pasteButton = this.driver.elementByXPath(
'//XCUIElementTypeMenuItem[@name="Paste"]'
);

await pasteButton.click();
await this.driver.sleep( 3000 ); // wait for paste notification to disappear
}

await toggleHtmlMode( this.driver, false );
}
Expand Down
Loading