Run Godot on travis as part of the tests

[touilleMan]

Following my post on #2641, I thing a good first start to add unit test in Godot would be to simply run the Godot binary on the CI platform once compiled.

I've encountered numerous trouble trying to do this on my side (see https://travis-ci.org/touilleMan/godot-python/jobs/351671519 and https://ci.appveyor.com/project/touilleMan/godot-python/build/1.0.241/job/q9j5tg176alem3y7)

• on travis/linux x11 server is not available, so I must rely on xvfb which doesn't seems to handle opengl 3.3 (even with LIBGL_ALWAYS_SOFTWARE=1)
• on appveyor/windows Godot starts but is frozen forever without outputing a single thing

I'm very aware those trouble could be caused by Pythonscript module (or by a misconfiguration), that's why having the command to run Godot on CI present (and executed !) on the official Godot repo would allow to narrow the possible source of errors.

[arthurtemple]

Regarding the X11 issue, I am unsure as to what the problem really is but would adding the --no-window option to platform x11 (it currently only works for Windows) be a solution?

[Calinou]

on travis/linux x11 server is not available, so I must rely on xvfb which doesn't seems to handle opengl 3.3 (even with LIBGL_ALWAYS_SOFTWARE=1)

You could compile and run the server platform instead of x11, since it doesn't require an X server to function. However, this means you cannot test editor functionality as server binaries essentially act as export templates.

[touilleMan]

@thorthur yep that would be a good solution I guess, but x11 without window would be roughly equivalent to the server target ?

[arthurtemple]

While server is roughly equivalent to x11 without window right now, I would like to point out:

1. "server" by itself denotes more a mode than a platform. This is debatable of course, see ongoing Implement headless (server) version for all desktop platforms #11389.
2. A possible use case could be to test with no graphics but with audio, in this case --no-window makes more sense.
3. There currently are slight differences in the initializations of platforms x11 and server, furthermore there is no guarantee those platforms will behave exactly the same with time (separate platform code). So there could be a need to test x11, and no server, because one might want to test a desktop release beforehand on a headless CI server.

@touilleMan If your use case does not belong to these points, I agree the --no-window argument is moot.

[cmfcmf]

We've been able to use both TravisCI and AppVeyor to automatically export our game.
You can find more information in our reddit post and our godot-export repository.

[Calinou]

After noticing #30630, I think it would make sense to use the headless binary we already build to export a simple project automatically. This way, we can catch regressions immediately. This project could either be committed here or in godot-tests. In the latter case, that repository would have to be cloned as part of the CI step. We'll need to make sure the editor always exits with a non-zero status code if there's any failure.

We could also try running a simple script using the -s command-line switch to check any regressions related to running standalone scripts.

I can work on this if this is desired.

[slapin]

My humble opinion is test data and test code should go with main repository for easier maintenance of consistency. But that is generic perspective, unless problems arise.

Regarding full blown testing see this: https://github.com/smartscenes/sstk/wiki/Headless-Rendering
https://github.com/stackgl/headless-gl/ but still solution is not perfect. I think full software implementation of DRM/OpenGL is more practical, also Vulkan testing will need some thought later.

[fire]

Having export being tested is already good, but it's common for the export to infinite loop on bugs.

[slapin]

can't travis time limit process execution?

[Calinou]

@slapin It already does so by default (jobs are limited to 1 hour), but we could use the timeout command and exit with a non-zero error code to restrict the execution time of a single command.

[slapin]

Global limit is a problem as it will fail job w/o running other jobs (which might indicate other errors and be useful), I think per job time limits were available which would allow other jobs still run while the renegade job will hit timeout...

…

On Thu, Jul 18, 2019 at 1:02 AM Hugo Locurcio ***@***.***> wrote: @slapin <https://github.com/slapin> It already does so by default (jobs are limited to 1 hour), but we could use the timeout command an exit with a non-zero error code to restrict the execution time of a single command. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17408?email_source=notifications&email_token=AAABPU5YCYY5MHQ7Z3OEXT3P76JGTA5CNFSM4EUVBU3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2GW3LY#issuecomment-512585135>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABPUZYDBPALNBKKX376V3P76JGTANCNFSM4EUVBU3A> .

[aaronfranke]

Is there any update on this issue? I've been thinking about possibly setting up Travis CI for the demo projects repository, to provide contributors with immediate feedback and to help with reviewing. Some static things I can think of include checking formatting of files (LF vs CRLF, BOM, EOL at EOF), checking C# files via clang-format, and maybe validating markdown of README files.

Aside from static checks, it would be amazing if I could set up Travis to run Godot and run some basic checks on the projects. For example, it would be nice to have Godot check all scenes and all GDScript files for warnings and errors, and possibly even run the projects to check for runtime errors.

[Xrayez]

I've already made some progress on this as described in godotengine/godot-tests#8, but since the 4.0 breakage the CI scripts and tests need to be updated again. 😛

[akien-mga]

Partially superseded by godotengine/godot-proposals#918, which would depend on godotengine/godot-proposals#991 to actually run unit tests on CI without having to rely on xvfb or similar.

If we want to extend this to also run actual Godot projects and do checks on visual output, etc., I would suggest opening a dedicated proposal specific to a project/scene-based testing framework for the Node API (as we discussed at Godot Sprint a few times), which can then be integrated in CI once available.

[Calinou]

For those stumbling upon this issue from a search engine, note that unit testing using doctest has been integrated into the Godot development process.