Use GHA with a custom runner to run automated tests #228

Josverl · 2022-04-20T10:53:23Z

Josverl
Apr 20, 2022

to improve quality ive started to set-up a simple hardware CI test

Pi4
- configured with DietPi
- installed GHA self hosted runner
- two micropython boards connected via USB
- created a CI workflow

Initial results:

Similar could be run on linux x64 , windows , and mac

additional work is needed to log the test results in a standard format (JUnitXML seems the common supported format) so that other GHA tools can work with it to provide better feedback and to join additional platforms.

Also currently the action-runner is only installed for my fork of the repo, but it can also be installed for an organization (PyCom)

Josverl · 2022-04-20T14:47:28Z

Josverl
Apr 20, 2022
Author

@jakobrosenberg ,
note that this also may help simplify validation of version dependencies.
It appears that some unit tests are failing on node 16,x which might be relevant as VSCode 1.66.2 is also based on node 16.

Version: 1.66.2 (user setup)
Electron: 17.2.0
Node.js: 16.13.0

0 replies

jakobrosenberg · 2022-04-21T06:11:36Z

jakobrosenberg
Apr 21, 2022

This is great!! I'll fix the Node 16 issue.

0 replies

jakobrosenberg · 2022-04-21T07:01:54Z

jakobrosenberg
Apr 21, 2022

Do you know if it's possible to connect a self hosted runner to Github? If so we could replicate your setup and have full integration tests for PRs.

1 reply

Josverl Apr 21, 2022
Author

Do you know if it's possible to connect a self hosted runner to Github? If so we could replicate your setup and have full integration tests for PRs.

Yes , that is what I have done,
I found it really much less complex than I expected, and it does not take much resources either.

I think it would make sense to run a few different GHA-SHRs with different setups and devices connected to them.
If you set this up we could work togetherby hooking up my runner to the pycom repo ( willl need to pass the setupkey securely )

The GH documentation is here :

https://docs.github.com/en/actions/hosting-your-own-runners

I have a small writeup of my setup that I will check-in to thenext-gha branch as well.

https://github.com/Josverl/pymakr-vsc/blob/next-gha/GHA-runner.md

jakobrosenberg · 2022-04-22T09:52:56Z

jakobrosenberg
Apr 22, 2022

I have solved the Node 16 issues locally. You can try the branch here https://github.com/pycom/pymakr-vsc/tree/node-16

If it works for you, I'll merge it into the next-staging branch.

6 replies

jakobrosenberg Apr 25, 2022

Great news! What kind of errors do you get with the integration test?

I'm currently rewriting the implementation. I'm hoping to get watch mode and interactive test filtering done by tomorrow. Can't make any promises though. Still a lot of gotchas when running the test runner inside the VSCode thread.

Josverl Apr 25, 2022
Author

note: I have a hunch that the failed test resolve mostly around :

the setup of the filesystem ( root = \ vs root = \flash )
MCU communication specifics ( ie send the reponse to p[rint("foo") in a single write versus multiple writes
possibly
MCU rest
the use of MCU specific micopython calls ?

it would be very good to have the test results in a structure (JUnit) test results document to find what tests are failing when

Josverl Apr 25, 2022
Author

What kind of errors do you get with the integration test?
Integration Log: https://gist.github.com/Josverl/ee539553fc31ad097d6d687aa1a3980d

Josverl Apr 25, 2022
Author

Is there a way to flash a pycom firmware to a non-pycom esp32 device ?
as I do not have any pycom devices that may be one of the causes of inconcistency

Josverl Apr 25, 2022
Author

I found a generic pycom firmware to test with a pycom stock filesystem
https://github.com/LIHUAWRITER/pycom-esp32-universal
(I could not find firmwares to use in testing on the pycom site , is there something similar ?)

Much better, only the print("foo") test fails

Tests passed: 24 / 25
Tests failed: 1 / 25

https://gist.github.com/Josverl/82ed4e7fae41b39d1ea1c4ee38c5351c

jakobrosenberg · 2022-04-26T06:20:26Z

jakobrosenberg
Apr 26, 2022

Much better, only the print("foo") test fails

So close now. Any idea what could be the cause of this? What happens if you print("foo") manually in the terminal?

1 reply

Josverl Apr 26, 2022
Author

I think the problem may be that:

the MCU send the response in multiple chunks , and not in a single string.
- chunck 1 : p
- chunck 2 : rint("foo")\r\n
- chunck 3 : 'foo
- chunck 4 : '\r\n>>>
that the function that is called in the test nextTerminalData(device) only recieves the first chunck or even stops after a single char

Integration test on Winx64:

 ❌ can use print command   13 ms
node_modules/probs/lib/reporters/consoleReporter/consoleReporter.js:12
AssertionError: 'p' == 'print("foo")\r\nfoo\r\n>>> ' AssertionError [ERR_ASSERTION]: 'p' == 'print("foo")\r\nfoo\r\n>>> '
	at TestInstance.callback (c:\develop\vscode\pymakr-vsc\test\suite\integration\devices.test.js:44:16)
	at async TestInstance.runTestCallback (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:76:13)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:120:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestFile.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestFile.js:28:9)
	at async QueueItem.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/utils/QueueManager.js:8:36)
	at async QueueManager.checkQueue (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/utils/QueueManager.js:41:13)
node_modules/probs/lib/reporters/consoleReporter/consoleReporter.js:14
          ✅ can disconnect    2 ms
no

Josverl · 2022-04-28T13:55:43Z

Josverl
Apr 28, 2022
Author

a quick update:
I gave Jest a try , first for the Unit tests.

Unit Tests

The Unit tests were very simple to update to run under Jest, with the follow pro's ::

Test explorer
run tests one-by-one / watch / only failed etc
reports in multiple formats, Junit, github action
code coverage report
raises exitcode=1 so GHA can detect failures

Integration tests
The integration tests are more a bit more work to integrate, mostly as there is no formal API for Jest

But there are a few templates in typescript that implement this, that need to be reworked into .js
this is work in progress

test explorer

reports in Github Actions

this simplifies spotting failing tests

there two tests are failing only on Mac and Node 14 or 16
-see the results of this run: https://github.com/Josverl/pymakr-vsc/actions/runs/2236176529

0 replies

jakobrosenberg · 2022-05-02T05:07:24Z

jakobrosenberg
May 2, 2022

the MCU send the response in multiple chunks , and not in a single string.
chunck 1 : p
chunck 2 : rint("foo")\r\n
chunck 3 : 'foo
chunck 4 : '\r\n>>>

I came across this too. It complicated detection of whether a device is running since the chunks can range from single characters, single lines and multiple lines. I think the affected test has been updated, so it might not affect you any more. If it does let me know and I'll have a look.

As for Jest, I started migrating my projects away from Jest last year because of scalability, performance and ESM issues. It's actually why I have started the probs project. Except for code coverage, probs has the same pros, though some are still in development. The lack of exitcode=1 was accidentally removed, but has been readded. I'll try get a release out asap. Apologies for the lack of documentation. This is on my todo list. Enabling watch mode (--watch) will provide a UI that lets you filter and watch tests.

1 reply

Josverl May 2, 2022
Author

I believe you that jest ain't perfect , but after switching from probs to jest:

I could run individual tests
found and fixed dependencies between these tests (no docs that probs also could do that )
I was able to find a few new issues.
create and use mocks

As there are other extension authors that also use jest , which allows for some re-use.
I don't really care which testing framework is used , as long as
P1:

it can test,
mock, spy
there is some level documentation ( even if its : most is like jest / jasmine / but not foo and bar )
P2:
report in a standard format ( to get structured reports via a tool such as testspace
preferably with IDE integration in VSCode

P3:

code coverage
there is some IP to reuse ( I can borrow from others working projects)

I know that jest won't solve all issues either, for example: Currently working on mocking sufficient of vscode to allow for unit testing of the devices and devices store in order to find and resolve the issues on windows.

Not sure what the way forward would be on this. Its not for me to decide, but currently the lack of mocking / spying / stubbing holds me back from contributing.

One way could be that probs and jest can both be used for unit testing with a naming convention & configuration to seperate)
Once probs has the needed/used capabilities functionality, I suppose it should be as easy to convert from jest back to probs as it was for the reverse.
To me that seems to be the most effective way to collaborate on PyMakr v-next

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use GHA with a custom runner to run automated tests #228

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Use GHA with a custom runner to run automated tests #228

Josverl Apr 20, 2022

Replies: 7 comments · 9 replies

Josverl Apr 20, 2022 Author

jakobrosenberg Apr 21, 2022

jakobrosenberg Apr 21, 2022

Josverl Apr 21, 2022 Author

jakobrosenberg Apr 22, 2022

jakobrosenberg Apr 25, 2022

Josverl Apr 25, 2022 Author

Josverl Apr 25, 2022 Author

Josverl Apr 25, 2022 Author

Josverl Apr 25, 2022 Author

jakobrosenberg Apr 26, 2022

Josverl Apr 26, 2022 Author

Josverl Apr 28, 2022 Author

Unit Tests

test explorer

reports in Github Actions

jakobrosenberg May 2, 2022

Josverl May 2, 2022 Author

Josverl
Apr 20, 2022

Replies: 7 comments 9 replies

Josverl
Apr 20, 2022
Author

jakobrosenberg
Apr 21, 2022

jakobrosenberg
Apr 21, 2022

Josverl Apr 21, 2022
Author

jakobrosenberg
Apr 22, 2022

Josverl Apr 25, 2022
Author

Josverl Apr 25, 2022
Author

Josverl Apr 25, 2022
Author

Josverl Apr 25, 2022
Author

jakobrosenberg
Apr 26, 2022

Josverl Apr 26, 2022
Author

Josverl
Apr 28, 2022
Author

jakobrosenberg
May 2, 2022

Josverl May 2, 2022
Author