Port robot framework tests to Playwright #3813

gforcada · 2023-06-28T07:55:01Z

STATUS

Report from Midsummer sprint 2023: playwright integration to plone.app.robotframework (via robotframework-browser) works! 🎉

All credits to @datakurre ✨

And even better, it not only works, but we don't have to have a sync day where all tests need to be converted at the same time, but rather we can move them incrementally!

TASKS

There are a plenty of things to do still, all arms and fingers are welcome! there are work items for everyone:

Please write your name next to a package if you want to work on porting them! 🙇🏾

PORTING TESTS

IMPORTANT NOTE: when porting tests from selenium to playwright, they have to be ported all tests from the same distribution in a single go, otherwise either the ported or the old ones will fail on CI.

Probably this is due to them sharing the same layer, if we keep the old test layer and create a new one for the playwright ones, all tests might pass.

RUN TESTS

To run tests locally:

get a buildout.coredev up and running
add the package you want to port to checkouts.cfg
run buildout again to get your package in development mode
run, once, ./bin/rfbrowser init
start porting the tests!
- ROBOT_BROWSER=chrome ./bin/test --all -s MY.PACKAGE -t .robot

DEBUGGING TESTS

By default playwright does not show the browser, as selenium used to do.

If you want to follow along though, add another environment variable ROBOT_DEBUG=true ✨

It will automatically stop (when using ROBOT_DEBUG) if a keyword fails 🤩 note though that if the keyword itself is wrong (an old selenium keyword) then it does not stop, it breaks the test and stops 😅

And as usual with @datakurre there is even more: VSCode integration right at your fingertips, run robot tests from within VSCode and run tests step by step 🤯

KEYWORDS STATUS

Notice that plenty of keywords that we were using for robot tests are no longer needed (the whole collection of Wait until... 💥 ) as playwright does implicit waits already 🚀

As soon as we have them running in CI we can see if we need to add them back, hopefully not 🍀 🤞🏾

robotframework-browser being much more younger than selenium has a smaller set of keywords available, see the p.a.discussion PR to get an idea on which replacements one can use.

On plone.app.robotframework we created quite a few custom ones. We either need to re-implement them with playwright based keywords, or drop them altogether, we need to see.

Exciting times! 🎉

The text was updated successfully, but these errors were encountered:

datakurre · 2023-06-29T15:29:30Z

ROBOT_TRACE=true will generate file named tracing deep under parts/tests/..., which could then be opened with bin/rfbrowser -F path/to/tracing show-trace for Playwright trace viewer.

These flags are defined in https://github.com/plone/plone.app.robotframework/blob/e5d98a98483729231a23cb5007e56b386e7001a6/src/plone/app/robotframework/browser.robot#L43 and are just the first way to integrate these Playwright and robotframework-browser features that came to mind. (robotsuite does the magic for passing any ROBOT_-prefixed environment variable into robot run as a robot variable)

gforcada · 2023-10-09T07:55:48Z

@thet @petschki I see that on plone.app.event there is a disabled robot test.

On my way from Bilbao to Barcelona with train I was looking at it, and it's a bit tricky to get it to work again, given that the date widget is browser dependent... should we try it anyway, or should it be removed entirely? 🤔

stevepiercy · 2024-01-21T20:08:38Z

For those following this issue, I created another issue in plone/documentation where I would like to leverage your work for Plone 6 Documentation for a Google Season of Docs proposal that I am working on.

plone/documentation#1611

1letter · 2024-09-13T09:42:39Z

@datakurre @gforcada what are the next steps if the tests are ported?

plone.themepreview is not core, is here the port really needed?

datakurre · 2024-09-13T13:03:04Z

plone.themepreview should be archived; I don't think it has been really used since very early Plone 5.It was used to "capture theme" with different devices through Sauce Labs, which uses Selenium.

mauritsvanrees · 2024-09-14T08:26:53Z

Please remind me: is './bin/rfbrowser init' only needed for playwright? Or can it also be useful for the old selenium based tests?

1letter · 2024-09-14T08:41:21Z

Please remind me: is './bin/rfbrowser init' only needed for playwright? Or can it also be useful for the old selenium based tests?

I think it's only needed for playwright based tests.

datakurre · 2024-09-14T12:08:33Z

Yes, it is only needed for playwright tests. Playwright is distributed through NPM and that command should imperatively install the required packages. Probably it also fetches the playwright compatible browser binaries.

mauritsvanrees · 2024-09-16T10:59:36Z

I am testing plone/plone.schemaeditor#118.

I have a problem on my Mac: bin/test hangs on the second test.

On buildout.coredev branch 6.1 I checked out plone.schemaeditor, did bin/rfbrowser init. With bin/test --all -s plone.schemaeditor, the tests hang on the second test.

I tried each test scenario individually with for example bin/test --all -s plone.schemaeditor -t "Add a content type" and they all worked fine.

But running two together leads to a hang:

$ bin/test --all -s plone.schemaeditor -t "Add a content type" -t "Delete field"
...
  Set up plone.schemaeditor.testing.ROBOT in 0.011 seconds.
  Set up plone.app.robotframework.remote.AutoLoginRemoteLibrary:RobotRemote in 0.000 seconds.
  Set up plone.testing.zope.WSGIServer in 0.038 seconds.
  Set up plone.schemaeditor.testing.ACCEPTANCE in 0.000 seconds.
  Running:
    1/2 (50.0%)/Users/maurits/shared-eggs/cp312/protobuf-4.24.4-py3.12.egg/google/protobuf/internal/well_known_types.py:93: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
  _EPOCH_DATETIME_NAIVE = datetime.datetime.utcfromtimestamp(0)
/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/playwright.py:57: ResourceWarning: unclosed file <_io.TextIOWrapper name='/Users/maurits/community/plone-coredev/6.1/parts/test/test_fields/Scenario_Add_a_content_type/playwright-log.txt' mode='w' encoding='UTF-8'>
  process = self.start_playwright()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/maurits/shared-eggs/cp312/grpcio-1.59.0-py3.12-macosx-14.2-x86_64.egg/grpc/_channel.py:1125: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  state.rpc_start_time = datetime.utcnow()
/Users/maurits/shared-eggs/cp312/grpcio-1.59.0-py3.12-macosx-14.2-x86_64.egg/grpc/_channel.py:226: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  state.rpc_end_time = datetime.utcnow()
<frozen importlib._bootstrap>:530: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
    2/2 (100.0%)

There it does nothing anymore.

I see a playwright log generated for the first test:

$ less .../parts/test/test_fields/Scenario_Add_a_content_type/playwright-log.txt
...
{"level":30,"time":"2024-09-16T10:56:32.535Z","pid":64385,"hostname":"MauBook19.local","msg":"Start of node method closeBrowser"}
{"level":30,"time":"2024-09-16T10:56:32.562Z","pid":64385,"hostname":"MauBook19.local","msg":"Removed page=8e96b647-b55f-43cc-bdab-d205d5b7d12c from context=4ce2a22d-b31b-44d0-ae80-89401eaad600 page stack"}
{"level":30,"time":"2024-09-16T10:56:32.935Z","pid":64385,"hostname":"MauBook19.local","msg":"End of node method closeBrowser"}
================= Original suppressed error =================
Error: Browser has been closed.
    at PlaywrightState.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9470:15)
    at PlaywrightServer.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9991:59)
    at PlaywrightServer.setTimeout (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:10237:57)
    at handleUnary (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/node_modules/@grpc/grpc-js/build/src/server.js:852:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
=============================================================
{"level":30,"time":"2024-09-16T10:56:32.944Z","pid":64385,"hostname":"MauBook19.local","msg":"Start of node method getBrowserCatalog"}
{"level":30,"time":"2024-09-16T10:56:32.944Z","pid":64385,"hostname":"MauBook19.local","msg":"End of node method getBrowserCatalog"}
================= Original suppressed error =================
Error: Browser has been closed.
    at PlaywrightState.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9470:15)
    at PlaywrightServer.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9991:59)
    at PlaywrightServer.setTimeout (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:10237:57)
    at handleUnary (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/node_modules/@grpc/grpc-js/build/src/server.js:852:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
=============================================================
{"level":30,"time":"2024-09-16T10:56:32.950Z","pid":64385,"hostname":"MauBook19.local","msg":"Start of node method getBrowserCatalog"}
{"level":30,"time":"2024-09-16T10:56:32.950Z","pid":64385,"hostname":"MauBook19.local","msg":"End of node method getBrowserCatalog"}

That sounds like teardown does not work correctly, and the second test cannot start. For the second test I only see an empty output.xml in the parts/test directory.

I have export ROBOT_BROWSER=headlesschrome in my bash profile, but see the same problem with ROBOT_BROWSER=chrome.

Any idea?

petschki · 2024-09-16T13:18:15Z

@mauritsvanrees I got this working on my Mac with the following version set:

robotframework-browser = 18.8.1
robotframework-assertion-engine = 3.0.3
robotframework-pythonlibcore = 4.4.1
grpcio-tools = 1.66.1

those versions are pinned here https://github.com/plone/buildout.coredev/blob/6.1/versions.cfg#L236-L239

NOTE: don't forget to rfbrowser init after upgrading those eggs

UPDATE:
these packages can be updated too

robotframework-debuglibrary = 2.5.0
prompt-toolkit = 3.0.47

Rotonen · 2024-09-16T16:42:31Z

Tangent: to make test runs platform independent (same locally as in pipeline), it could be prudent to provide devcontainers.

https://containers.dev/

mauritsvanrees · 2024-09-17T08:59:44Z

Thanks @petschki, with those pins it works for me. I have merged your coredev PR.

Could you check if those versions work on 6.0 as well?

mauritsvanrees · 2024-09-17T11:39:53Z

@gforcada The Jenkins nodes may need some upgrade. "Host system is missing dependencies to run browsers." See plone/buildout.coredev#948 (comment)

gforcada · 2024-09-18T09:13:15Z

Sorry for the delay on replies, I'm rather busy as of late, hopefully in 2 weeks the big project I work on will be deployed (and the more bugs and features will be asked 😆 ) and I will have some time 🤞🏾

Ok, I found some time, I bumped nodejs to v20.5.1

gforcada · 2024-09-18T09:18:21Z

Turns out that 20.5.1 is not enough 🙃

petschki · 2024-09-18T09:28:37Z

Log says

required: { node: '^18.18.0 || ^20.9.0 || >=21.1.0' }

petschki · 2024-09-18T09:49:59Z

And chromedriver seems to be outdated too. See https://jenkins.plone.org/job/pull-request-6.1-3.12/614/consoleFull ...

gforcada · 2024-09-19T05:41:49Z

So, finally, in a sleepless night I found the time to look at the problem 🎉

I installed nvm with node version 22, and enabled nvm to be reachable on jenkins jobs.

The long list of dependencies can be installed (if anyone hits the same problem) with npx playwright install-deps ✨

Oh, and indeed @petschki chromedriver was again outdated, fortunately that's easier to fix (and it is 😄 )

/cc @mauritsvanrees @1letter (I'm not sure in which other PR/issue we were also discussing CI problems of robot tests 😓 )

Seems that robot tests run again: https://jenkins.plone.org/job/plone-6.1-python-3.12-robot-chrome/561/console

mauritsvanrees · 2024-09-19T06:12:50Z

Thanks a lot! And I hope you sleep better next night. 😴💫

1letter · 2024-09-19T12:09:37Z

@gforcada Sorry, but an error happen on install rfbrowser

https://jenkins.plone.org/job/pull-request-6.1-3.10/360/console

+ bin/rfbrowser init
2024-09-19 04:55:01,402 [INFO    ] ==============================================================================================================
2024-09-19 04:55:01,402 [INFO    ] Installing node dependencies...
2024-09-19 04:55:01,403 [INFO    ] Couldn't execute npm. Please ensure you have node.js and npm installed and in PATH.See https://nodejs.org/ for documentation
2024-09-19 04:55:01,403 [INFO    ] ==============================================================================================================
2024-09-19 04:55:01,403 [INFO    ] Traceback (most recent call last):
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 382, in init
    _rfbrowser_init(skip_browsers, silent_mode, with_deps, browser)
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 155, in _rfbrowser_init
    _check_npm()
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 147, in _check_npm
    raise exception
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 135, in _check_npm
    subprocess.run(
  File "/srv/python3.10/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/srv/python3.10/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/srv/python3.10/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'npm'

gforcada · 2024-10-12T22:03:04Z

Sorry again, my life is (a bit) in shambles, but I think, at least on Node1, robot tests do work, at least the bin/rfbrowser init does pass:

https://jenkins.plone.org/job/plone-6.0-python-3.11-robot-chrome/1457/console

🤞🏾

mauritsvanrees · 2024-10-21T10:53:12Z

Awesome!
Thanks a lot @1letter, @petschki, @gforcada !

petschki mentioned this issue Jun 29, 2023

Port robot tests from plone.app.widgets and update to Playwright integration plone/plone.app.z3cform#174

Closed

This was referenced Jun 30, 2023

Run playwright based tests plone/meta#155

Closed

Enable or remove robot tests collective/collective.glossary#66

Open

gforcada mentioned this issue Jul 9, 2023

On Plone 6 initialize playwright plone/jenkins.plone.org#341

Merged

This was referenced Oct 5, 2023

Make robot tests more stable/robust #3723

Closed

Port robot tests plone/plone.app.contenttypes#673

Closed

Port robot tests plone/plone.app.dexterity#377

Closed

thet added the 99 tag: bssp Potential task for the Buschenschanksprint label Apr 24, 2024

thet added this to Buschenschanksprint 2024 Apr 24, 2024

thet moved this to Todo in Buschenschanksprint 2024 Apr 24, 2024

This was referenced Sep 9, 2024

port robot tests plone/plone.app.event#408

Closed

Playwright based robot test plone/plone.app.event#409

Merged

port robot tests plone/plone.app.multilingual#459

Closed

port robot tests plone/plone.app.multilingual#460

Merged

1letter mentioned this issue Sep 13, 2024

port robot tests plone/plone.schemaeditor#117

Closed

This was referenced Sep 14, 2024

port robot tests plone/plone.schemaeditor#118

Merged

port robot tests plone/plone.app.dexterity#393

Closed

Port robot test plone/plone.app.dexterity#394

Merged

This was referenced Sep 18, 2024

Port robot test plone/plone.app.contenttypes#697

Closed

port robot tests plone/plone.app.contenttypes#698

Closed

1letter removed the 99 tag: bssp Potential task for the Buschenschanksprint label Sep 24, 2024

1letter mentioned this issue Oct 7, 2024

Port robot tests #4019

Merged

27 tasks

1letter linked a pull request Oct 7, 2024 that will close this issue

Port robot tests #4019

Merged

27 tasks

This was referenced Oct 8, 2024

Port tests - really needed? collective/robotsuite#29

Closed

Port robot tests plone/plone.app.contenttypes#704

Merged

davisagli mentioned this issue Oct 16, 2024

Question for write a Test with pytest in Plone collective/pytest-plone#16

Open

petschki closed this as completed in #4019 Oct 18, 2024

github-project-automation bot moved this from Todo to Done in Buschenschanksprint 2024 Oct 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port robot framework tests to Playwright #3813

Port robot framework tests to Playwright #3813

gforcada commented Jun 28, 2023 •

edited

Loading

datakurre commented Jun 29, 2023

gforcada commented Oct 9, 2023

stevepiercy commented Jan 21, 2024

1letter commented Sep 13, 2024

datakurre commented Sep 13, 2024

mauritsvanrees commented Sep 14, 2024

1letter commented Sep 14, 2024

datakurre commented Sep 14, 2024

mauritsvanrees commented Sep 16, 2024

petschki commented Sep 16, 2024 •

edited

Loading

Rotonen commented Sep 16, 2024

mauritsvanrees commented Sep 17, 2024

mauritsvanrees commented Sep 17, 2024

gforcada commented Sep 18, 2024

gforcada commented Sep 18, 2024

petschki commented Sep 18, 2024

petschki commented Sep 18, 2024

gforcada commented Sep 19, 2024

mauritsvanrees commented Sep 19, 2024

1letter commented Sep 19, 2024

gforcada commented Oct 12, 2024

mauritsvanrees commented Oct 21, 2024

Port robot framework tests to Playwright #3813

Port robot framework tests to Playwright #3813

Comments

gforcada commented Jun 28, 2023 • edited Loading

STATUS

TASKS

PORTING TESTS

RUN TESTS

DEBUGGING TESTS

KEYWORDS STATUS

datakurre commented Jun 29, 2023

gforcada commented Oct 9, 2023

stevepiercy commented Jan 21, 2024

1letter commented Sep 13, 2024

datakurre commented Sep 13, 2024

mauritsvanrees commented Sep 14, 2024

1letter commented Sep 14, 2024

datakurre commented Sep 14, 2024

mauritsvanrees commented Sep 16, 2024

petschki commented Sep 16, 2024 • edited Loading

Rotonen commented Sep 16, 2024

mauritsvanrees commented Sep 17, 2024

mauritsvanrees commented Sep 17, 2024

gforcada commented Sep 18, 2024

gforcada commented Sep 18, 2024

petschki commented Sep 18, 2024

petschki commented Sep 18, 2024

gforcada commented Sep 19, 2024

mauritsvanrees commented Sep 19, 2024

1letter commented Sep 19, 2024

gforcada commented Oct 12, 2024

mauritsvanrees commented Oct 21, 2024

gforcada commented Jun 28, 2023 •

edited

Loading

petschki commented Sep 16, 2024 •

edited

Loading