Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2E testing: Use JSON serialization for getAllBlocks #31199

Merged
merged 3 commits into from
Apr 27, 2021

Conversation

david-szabo97
Copy link
Member

@david-szabo97 david-szabo97 commented Apr 26, 2021

Description

page.evaluate used in wpDataSelect, returns undefined when we return a non-serializable value from the evaluation. Non-serializable values are function, DOM element etc.

Reference:
https://pptr.dev/#?product=Puppeteer&version=v9.0.0&show=api-pageevaluatepagefunction-args

If the function passed to the page.evaluate returns a non-Serializable value, then page.evaluate resolves to undefined. DevTools Protocol also supports transferring some additional values that are not serializable by JSON: -0, NaN, Infinity, -Infinity, and bigint literals.

For a real world example, we store validationIssues in the block-editor store for each block. When we have a validation issue, we store the ref and owner (DOM element handles). Those are non-serializable so the evaluation returns with undefined. Which is causing the tests to fail.

As a fix, we turn the result into a JSON string, then we parse it. JSON replaces all unserializable values with null, so the evaluation can return safely.

Reference:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify

undefined, Functions, and Symbols are not valid JSON values. If any such values are encountered during conversion they are either omitted (when found in an object) or changed to null (when found in an array). JSON.stringify() can return undefined when passing in "pure" values like JSON.stringify(function(){}) or JSON.stringify(undefined).

This PR ensures we are returning an object that is serializable by Puppeteer.

How has this been tested?

Tests are passing

Types of changes

E2E tests consistency

Checklist:

  • My code is tested.
  • My code follows the WordPress code style.
  • My code follows the accessibility standards.
  • I've tested my changes with keyboard and screen readers.
  • My code has proper inline documentation.
  • I've included developer documentation if appropriate.
  • I've updated all React Native files affected by any refactorings/renamings in this PR (please manually search all *.native.js files for terms that need renaming or removal).

@david-szabo97 david-szabo97 self-assigned this Apr 26, 2021
@david-szabo97 david-szabo97 added [Tool] E2E Test Utils /packages/e2e-test-utils [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. labels Apr 26, 2021
@github-actions
Copy link

github-actions bot commented Apr 26, 2021

Size Change: -4 B (0%)

Total Size: 1.39 MB

Filename Size Change
build/block-editor/style.css 13 kB -1 B (0%)
build/block-library/blocks/template-part/editor-rtl.css 551 B -1 B (0%)
build/block-library/blocks/template-part/editor.css 550 B -1 B (0%)
build/block-library/editor-rtl.css 9.4 kB -1 B (0%)
ℹ️ View Unchanged
Filename Size Change
build/a11y/index.js 1.12 kB 0 B
build/annotations/index.js 2.93 kB 0 B
build/api-fetch/index.js 2.42 kB 0 B
build/autop/index.js 2.28 kB 0 B
build/blob/index.js 673 B 0 B
build/block-directory/index.js 6.6 kB 0 B
build/block-directory/style-rtl.css 993 B 0 B
build/block-directory/style.css 995 B 0 B
build/block-editor/index.js 116 kB 0 B
build/block-editor/style-rtl.css 13 kB 0 B
build/block-library/blocks/archives/editor-rtl.css 61 B 0 B
build/block-library/blocks/archives/editor.css 60 B 0 B
build/block-library/blocks/audio/editor-rtl.css 58 B 0 B
build/block-library/blocks/audio/editor.css 58 B 0 B
build/block-library/blocks/audio/style-rtl.css 112 B 0 B
build/block-library/blocks/audio/style.css 112 B 0 B
build/block-library/blocks/block/editor-rtl.css 161 B 0 B
build/block-library/blocks/block/editor.css 161 B 0 B
build/block-library/blocks/button/editor-rtl.css 475 B 0 B
build/block-library/blocks/button/editor.css 474 B 0 B
build/block-library/blocks/button/style-rtl.css 501 B 0 B
build/block-library/blocks/button/style.css 500 B 0 B
build/block-library/blocks/buttons/editor-rtl.css 315 B 0 B
build/block-library/blocks/buttons/editor.css 315 B 0 B
build/block-library/blocks/buttons/style-rtl.css 368 B 0 B
build/block-library/blocks/buttons/style.css 368 B 0 B
build/block-library/blocks/calendar/style-rtl.css 208 B 0 B
build/block-library/blocks/calendar/style.css 208 B 0 B
build/block-library/blocks/categories/editor-rtl.css 84 B 0 B
build/block-library/blocks/categories/editor.css 83 B 0 B
build/block-library/blocks/categories/style-rtl.css 79 B 0 B
build/block-library/blocks/categories/style.css 79 B 0 B
build/block-library/blocks/code/style-rtl.css 90 B 0 B
build/block-library/blocks/code/style.css 90 B 0 B
build/block-library/blocks/columns/editor-rtl.css 190 B 0 B
build/block-library/blocks/columns/editor.css 190 B 0 B
build/block-library/blocks/columns/style-rtl.css 422 B 0 B
build/block-library/blocks/columns/style.css 422 B 0 B
build/block-library/blocks/cover/editor-rtl.css 603 B 0 B
build/block-library/blocks/cover/editor.css 604 B 0 B
build/block-library/blocks/cover/style-rtl.css 1.22 kB 0 B
build/block-library/blocks/cover/style.css 1.22 kB 0 B
build/block-library/blocks/embed/editor-rtl.css 486 B 0 B
build/block-library/blocks/embed/editor.css 486 B 0 B
build/block-library/blocks/embed/style-rtl.css 401 B 0 B
build/block-library/blocks/embed/style.css 400 B 0 B
build/block-library/blocks/file/editor-rtl.css 301 B 0 B
build/block-library/blocks/file/editor.css 300 B 0 B
build/block-library/blocks/file/frontend.js 772 B 0 B
build/block-library/blocks/file/style-rtl.css 255 B 0 B
build/block-library/blocks/file/style.css 255 B 0 B
build/block-library/blocks/freeform/editor-rtl.css 2.45 kB 0 B
build/block-library/blocks/freeform/editor.css 2.45 kB 0 B
build/block-library/blocks/gallery/editor-rtl.css 704 B 0 B
build/block-library/blocks/gallery/editor.css 705 B 0 B
build/block-library/blocks/gallery/style-rtl.css 1.06 kB 0 B
build/block-library/blocks/gallery/style.css 1.05 kB 0 B
build/block-library/blocks/group/editor-rtl.css 160 B 0 B
build/block-library/blocks/group/editor.css 160 B 0 B
build/block-library/blocks/group/style-rtl.css 57 B 0 B
build/block-library/blocks/group/style.css 57 B 0 B
build/block-library/blocks/heading/editor-rtl.css 129 B 0 B
build/block-library/blocks/heading/editor.css 129 B 0 B
build/block-library/blocks/heading/style-rtl.css 76 B 0 B
build/block-library/blocks/heading/style.css 76 B 0 B
build/block-library/blocks/html/editor-rtl.css 281 B 0 B
build/block-library/blocks/html/editor.css 281 B 0 B
build/block-library/blocks/image/editor-rtl.css 717 B 0 B
build/block-library/blocks/image/editor.css 716 B 0 B
build/block-library/blocks/image/style-rtl.css 476 B 0 B
build/block-library/blocks/image/style.css 478 B 0 B
build/block-library/blocks/latest-comments/style-rtl.css 281 B 0 B
build/block-library/blocks/latest-comments/style.css 282 B 0 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B 0 B
build/block-library/blocks/latest-posts/editor.css 137 B 0 B
build/block-library/blocks/latest-posts/style-rtl.css 523 B 0 B
build/block-library/blocks/latest-posts/style.css 522 B 0 B
build/block-library/blocks/legacy-widget/editor-rtl.css 398 B 0 B
build/block-library/blocks/legacy-widget/editor.css 399 B 0 B
build/block-library/blocks/list/style-rtl.css 63 B 0 B
build/block-library/blocks/list/style.css 63 B 0 B
build/block-library/blocks/media-text/editor-rtl.css 176 B 0 B
build/block-library/blocks/media-text/editor.css 176 B 0 B
build/block-library/blocks/media-text/style-rtl.css 492 B 0 B
build/block-library/blocks/media-text/style.css 489 B 0 B
build/block-library/blocks/more/editor-rtl.css 434 B 0 B
build/block-library/blocks/more/editor.css 434 B 0 B
build/block-library/blocks/navigation-link/editor-rtl.css 597 B 0 B
build/block-library/blocks/navigation-link/editor.css 597 B 0 B
build/block-library/blocks/navigation-link/style-rtl.css 1.17 kB 0 B
build/block-library/blocks/navigation-link/style.css 1.17 kB 0 B
build/block-library/blocks/navigation/editor-rtl.css 1.19 kB 0 B
build/block-library/blocks/navigation/editor.css 1.19 kB 0 B
build/block-library/blocks/navigation/style-rtl.css 272 B 0 B
build/block-library/blocks/navigation/style.css 271 B 0 B
build/block-library/blocks/nextpage/editor-rtl.css 395 B 0 B
build/block-library/blocks/nextpage/editor.css 395 B 0 B
build/block-library/blocks/page-list/editor-rtl.css 239 B 0 B
build/block-library/blocks/page-list/editor.css 240 B 0 B
build/block-library/blocks/page-list/style-rtl.css 167 B 0 B
build/block-library/blocks/page-list/style.css 167 B 0 B
build/block-library/blocks/paragraph/editor-rtl.css 157 B 0 B
build/block-library/blocks/paragraph/editor.css 157 B 0 B
build/block-library/blocks/paragraph/style-rtl.css 247 B 0 B
build/block-library/blocks/paragraph/style.css 248 B 0 B
build/block-library/blocks/post-author/editor-rtl.css 209 B 0 B
build/block-library/blocks/post-author/editor.css 209 B 0 B
build/block-library/blocks/post-author/style-rtl.css 183 B 0 B
build/block-library/blocks/post-author/style.css 184 B 0 B
build/block-library/blocks/post-comments-form/style-rtl.css 250 B 0 B
build/block-library/blocks/post-comments-form/style.css 250 B 0 B
build/block-library/blocks/post-content/editor-rtl.css 139 B 0 B
build/block-library/blocks/post-content/editor.css 139 B 0 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B 0 B
build/block-library/blocks/post-excerpt/editor.css 73 B 0 B
build/block-library/blocks/post-excerpt/style-rtl.css 69 B 0 B
build/block-library/blocks/post-excerpt/style.css 69 B 0 B
build/block-library/blocks/post-featured-image/editor-rtl.css 338 B 0 B
build/block-library/blocks/post-featured-image/editor.css 338 B 0 B
build/block-library/blocks/post-featured-image/style-rtl.css 100 B 0 B
build/block-library/blocks/post-featured-image/style.css 100 B 0 B
build/block-library/blocks/post-title/style-rtl.css 60 B 0 B
build/block-library/blocks/post-title/style.css 60 B 0 B
build/block-library/blocks/preformatted/style-rtl.css 103 B 0 B
build/block-library/blocks/preformatted/style.css 103 B 0 B
build/block-library/blocks/pullquote/editor-rtl.css 183 B 0 B
build/block-library/blocks/pullquote/editor.css 183 B 0 B
build/block-library/blocks/pullquote/style-rtl.css 318 B 0 B
build/block-library/blocks/pullquote/style.css 318 B 0 B
build/block-library/blocks/query-loop/editor-rtl.css 83 B 0 B
build/block-library/blocks/query-loop/editor.css 82 B 0 B
build/block-library/blocks/query-loop/style-rtl.css 315 B 0 B
build/block-library/blocks/query-loop/style.css 317 B 0 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B 0 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B 0 B
build/block-library/blocks/query-pagination/editor-rtl.css 270 B 0 B
build/block-library/blocks/query-pagination/editor.css 262 B 0 B
build/block-library/blocks/query-pagination/style-rtl.css 168 B 0 B
build/block-library/blocks/query-pagination/style.css 168 B 0 B
build/block-library/blocks/query-title/editor-rtl.css 86 B 0 B
build/block-library/blocks/query-title/editor.css 86 B 0 B
build/block-library/blocks/query/editor-rtl.css 131 B 0 B
build/block-library/blocks/query/editor.css 132 B 0 B
build/block-library/blocks/quote/style-rtl.css 169 B 0 B
build/block-library/blocks/quote/style.css 169 B 0 B
build/block-library/blocks/rss/editor-rtl.css 201 B 0 B
build/block-library/blocks/rss/editor.css 202 B 0 B
build/block-library/blocks/rss/style-rtl.css 290 B 0 B
build/block-library/blocks/rss/style.css 290 B 0 B
build/block-library/blocks/search/editor-rtl.css 189 B 0 B
build/block-library/blocks/search/editor.css 189 B 0 B
build/block-library/blocks/search/style-rtl.css 359 B 0 B
build/block-library/blocks/search/style.css 362 B 0 B
build/block-library/blocks/separator/editor-rtl.css 99 B 0 B
build/block-library/blocks/separator/editor.css 99 B 0 B
build/block-library/blocks/separator/style-rtl.css 251 B 0 B
build/block-library/blocks/separator/style.css 251 B 0 B
build/block-library/blocks/shortcode/editor-rtl.css 512 B 0 B
build/block-library/blocks/shortcode/editor.css 512 B 0 B
build/block-library/blocks/site-logo/editor-rtl.css 440 B 0 B
build/block-library/blocks/site-logo/editor.css 441 B 0 B
build/block-library/blocks/site-logo/style-rtl.css 154 B 0 B
build/block-library/blocks/site-logo/style.css 154 B 0 B
build/block-library/blocks/social-link/editor-rtl.css 164 B 0 B
build/block-library/blocks/social-link/editor.css 165 B 0 B
build/block-library/blocks/social-links/editor-rtl.css 796 B 0 B
build/block-library/blocks/social-links/editor.css 795 B 0 B
build/block-library/blocks/social-links/style-rtl.css 1.32 kB 0 B
build/block-library/blocks/social-links/style.css 1.33 kB 0 B
build/block-library/blocks/spacer/editor-rtl.css 308 B 0 B
build/block-library/blocks/spacer/editor.css 308 B 0 B
build/block-library/blocks/spacer/style-rtl.css 48 B 0 B
build/block-library/blocks/spacer/style.css 48 B 0 B
build/block-library/blocks/table/editor-rtl.css 478 B 0 B
build/block-library/blocks/table/editor.css 478 B 0 B
build/block-library/blocks/table/style-rtl.css 407 B 0 B
build/block-library/blocks/table/style.css 407 B 0 B
build/block-library/blocks/tag-cloud/editor-rtl.css 118 B 0 B
build/block-library/blocks/tag-cloud/editor.css 118 B 0 B
build/block-library/blocks/tag-cloud/style-rtl.css 94 B 0 B
build/block-library/blocks/tag-cloud/style.css 94 B 0 B
build/block-library/blocks/term-description/editor-rtl.css 90 B 0 B
build/block-library/blocks/term-description/editor.css 90 B 0 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B 0 B
build/block-library/blocks/text-columns/editor.css 95 B 0 B
build/block-library/blocks/text-columns/style-rtl.css 166 B 0 B
build/block-library/blocks/text-columns/style.css 166 B 0 B
build/block-library/blocks/verse/style-rtl.css 87 B 0 B
build/block-library/blocks/verse/style.css 87 B 0 B
build/block-library/blocks/video/editor-rtl.css 569 B 0 B
build/block-library/blocks/video/editor.css 570 B 0 B
build/block-library/blocks/video/style-rtl.css 169 B 0 B
build/block-library/blocks/video/style.css 169 B 0 B
build/block-library/common-rtl.css 1.31 kB 0 B
build/block-library/common.css 1.31 kB 0 B
build/block-library/editor.css 9.4 kB 0 B
build/block-library/index.js 142 kB 0 B
build/block-library/reset-rtl.css 506 B 0 B
build/block-library/reset.css 507 B 0 B
build/block-library/style-rtl.css 9.46 kB 0 B
build/block-library/style.css 9.46 kB 0 B
build/block-library/theme-rtl.css 692 B 0 B
build/block-library/theme.css 693 B 0 B
build/block-serialization-default-parser/index.js 1.3 kB 0 B
build/block-serialization-spec-parser/index.js 3.06 kB 0 B
build/blocks/index.js 46.6 kB 0 B
build/components/index.js 275 kB 0 B
build/components/style-rtl.css 16.3 kB 0 B
build/components/style.css 16.3 kB 0 B
build/compose/index.js 9.97 kB 0 B
build/core-data/index.js 12 kB 0 B
build/customize-widgets/index.js 5.79 kB 0 B
build/customize-widgets/style-rtl.css 666 B 0 B
build/customize-widgets/style.css 667 B 0 B
build/data-controls/index.js 826 B 0 B
build/data/index.js 7.55 kB 0 B
build/date/index.js 31.8 kB 0 B
build/deprecated/index.js 738 B 0 B
build/dom-ready/index.js 577 B 0 B
build/dom/index.js 4.61 kB 0 B
build/edit-navigation/index.js 13.3 kB 0 B
build/edit-navigation/style-rtl.css 2.84 kB 0 B
build/edit-navigation/style.css 2.85 kB 0 B
build/edit-post/classic-rtl.css 454 B 0 B
build/edit-post/classic.css 454 B 0 B
build/edit-post/index.js 335 kB 0 B
build/edit-post/style-rtl.css 6.95 kB 0 B
build/edit-post/style.css 6.93 kB 0 B
build/edit-site/index.js 25.2 kB 0 B
build/edit-site/style-rtl.css 4.9 kB 0 B
build/edit-site/style.css 4.89 kB 0 B
build/edit-widgets/index.js 13 kB 0 B
build/edit-widgets/style-rtl.css 2.97 kB 0 B
build/edit-widgets/style.css 2.98 kB 0 B
build/editor/index.js 60.5 kB 0 B
build/editor/style-rtl.css 3.9 kB 0 B
build/editor/style.css 3.9 kB 0 B
build/element/index.js 3.44 kB 0 B
build/escape-html/index.js 739 B 0 B
build/format-library/index.js 5.68 kB 0 B
build/format-library/style-rtl.css 637 B 0 B
build/format-library/style.css 639 B 0 B
build/hooks/index.js 1.76 kB 0 B
build/html-entities/index.js 628 B 0 B
build/i18n/index.js 3.73 kB 0 B
build/is-shallow-equal/index.js 710 B 0 B
build/keyboard-shortcuts/index.js 1.65 kB 0 B
build/keycodes/index.js 1.43 kB 0 B
build/list-reusable-blocks/index.js 2.07 kB 0 B
build/list-reusable-blocks/style-rtl.css 629 B 0 B
build/list-reusable-blocks/style.css 628 B 0 B
build/media-utils/index.js 3.08 kB 0 B
build/notices/index.js 1.07 kB 0 B
build/nux/index.js 2.31 kB 0 B
build/nux/style-rtl.css 718 B 0 B
build/nux/style.css 716 B 0 B
build/plugins/index.js 2 kB 0 B
build/primitives/index.js 1.03 kB 0 B
build/priority-queue/index.js 791 B 0 B
build/react-i18n/index.js 924 B 0 B
build/redux-routine/index.js 2.82 kB 0 B
build/reusable-blocks/index.js 2.56 kB 0 B
build/reusable-blocks/style-rtl.css 225 B 0 B
build/reusable-blocks/style.css 225 B 0 B
build/rich-text/index.js 11.7 kB 0 B
build/server-side-render/index.js 1.64 kB 0 B
build/shortcode/index.js 1.69 kB 0 B
build/token-list/index.js 847 B 0 B
build/url/index.js 1.95 kB 0 B
build/viewport/index.js 1.28 kB 0 B
build/warning/index.js 1.14 kB 0 B
build/wordcount/index.js 1.24 kB 0 B

compressed-size-action

@david-szabo97 david-szabo97 changed the title E2E testing: Use JSON serialization for wpDataSelect E2E testing: Use JSON serialization for getAllBlocks Apr 26, 2021
Copy link
Contributor

@ntsekouras ntsekouras left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks!

@david-szabo97
Copy link
Member Author

The PR is scoped down to only affect getAllBlocks. My first thought was to add this to wpDataSelect, which would affect many more tests. I ran the test suite and modifying wpDataSelect is also making the tests pass. It's a more future-proof solution.

I wonder what others think, should we add this to wpDataSelect or keep it scoped to getAllBlocks?

@jeyip
Copy link
Contributor

jeyip commented Apr 26, 2021

My first thought was to add this to wpDataSelect, which would affect many more tests. I ran the test suite and modifying wpDataSelect is also making the tests pass. It's a more future-proof solution.

I'd agree with your intuition here. I only see wpDataSelect being used in 6 files, which seems like a relatively small surface area to manage.

  1. packages/e2e-test-utils/src/are-pre-publish-checks-enabled.js
  2. packages/e2e-test-utils/src/get-all-blocks.js
  3. packages/e2e-test-utils/src/get-current-post-content.js
  4. packages/e2e-test-utils/src/get-edited-post-content.js
  5. packages/e2e-test-utils/src/index.js
  6. packages/e2e-tests/experimental-features.js

I can also imagine others running into the same problem in the future, so I'd vote for making the change to wpDataSelect itself. Worse comes to worse, we'd break e2e testing for a short period of time, at which point we'd revert these relatively compact, non-user facing changes.

Copy link
Contributor

@jeyip jeyip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current changes look good to me too.

@jeyip
Copy link
Contributor

jeyip commented Apr 26, 2021

NB: If we end up merging changes scoped only to getAllBlocks, it'd be great to have a follow-up ticket to update other files that reference wpDataSelect

@ntsekouras
Copy link
Contributor

I wonder what others think, should we add this to wpDataSelect or keep it scoped to getAllBlocks?

I think we can restrict this to getAllBlocks for now that has problems with block deprecations. Running parse and stringify for every wp.data.select call might not be needed and definitely is not needed in GB repo now.

We should not forget that both are public APIs and are used outside of GB as well. Since there are no created issues for either of them, IMO it's okay to land the only needed change for getAllBlocks.

@jeyip
Copy link
Contributor

jeyip commented Apr 26, 2021

We should not forget that both are public APIs and are used outside of GB as well. Since there are no created issues for either of them, IMO it's okay to land the only needed change for getAllBlocks.

That makes sense 👍 .

In that case, could we leave more explicit documentation somewhere? It seems easy to overlook the fact that wpDataSelect will only behave as expected if returning a serializable value. It might read something like "If using the wpDataSelect e2e helper, then ensure that the return value from the selector is serializable (e.g. NOT function, Dom element, etc.)."

@ntsekouras
Copy link
Contributor

ntsekouras commented Apr 27, 2021

In that case, could we leave more explicit documentation somewhere? It seems easy to overlook the fact that wpDataSelect will only behave as expected if returning a serializable value. It might read something like "If using the wpDataSelect e2e helper, then ensure that the return value from the selector is serializable (e.g. function, Dom element, etc.)."

Agreed! We can do this in a follow up.

I'll merge this though now, as is needed for #30804

Thanks for all the work here!

@ntsekouras ntsekouras merged commit 40c1dc3 into trunk Apr 27, 2021
@ntsekouras ntsekouras deleted the add/wp-data-select-serialization branch April 27, 2021 06:50
@github-actions github-actions bot added this to the Gutenberg 10.6 milestone Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Tool] E2E Test Utils /packages/e2e-test-utils [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants