Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add runner for GDScript testing #47701

Merged
merged 3 commits into from
Apr 16, 2021
Merged

Conversation

vnen
Copy link
Member

@vnen vnen commented Apr 7, 2021

This is meant for testing the GDScript implementation, not for testing user scripts nor testing the engine using scripts.

Tests consists in a GDScript file and a .out file with the expected output. The .out file format is: expected status (based on the enum GDScriptTest::TestStatus) on the first line, followed by either an error message or the resulting output. Warnings are added after the first line, before the output (or compiler errors) if the parser pass without any error.

The test script must have a function called test() which takes no argument. Such function will be called by the test runner. The test should not have any dependency unless it's part of the test too. Global classes (using class_name) are registered before the runner starts, so those should work if needed.

Use the command godot --gdscript-generate-tests godot-source/modules/gdscript/tests/scripts to update the .out files
with the current output (make sure the output are the expected values before committing).

The tests themselves are part of the doctest suite so those can be executed with godot --test.


Thanks @Xrayez for the initial work porting the runner to doctest. This therefore supersedes #41074.

I moved the test suite to its own file because otherwise the tests would run twice.

@vnen vnen added this to the 4.0 milestone Apr 7, 2021
@vnen vnen requested review from a team as code owners April 7, 2021 13:50
tests/test_main.cpp Outdated Show resolved Hide resolved
@vnen vnen force-pushed the gdscript-test-runner branch 2 times, most recently from 5b6e116 to fc2f19f Compare April 7, 2021 17:41
@vnen vnen requested a review from a team as a code owner April 7, 2021 17:41
@vnen
Copy link
Member Author

vnen commented Apr 7, 2021

Update to solve a couple of issues (separate commits since not directly related to this).

vnen and others added 3 commits April 8, 2021 09:34
This is meant for testing the GDScript implementation, not for testing
user scripts nor testing the engine using scripts.

Tests consists in a GDScript file and a .out file with the expected
output. The .out file format is: expected status (based on the enum
GDScriptTest::TestStatus) on the first line, followed by either an error
message or the resulting output. Warnings are added after the first
line, before the output (or compiler errors) if the parser pass without
any error.

The test script must have a function called `test()` which takes no
argument. Such function will be called by the test runner. The test
should not have any dependency unless it's part of the test too. Global
classes (using `class_name`) are registered before the runner starts, so
those should work if needed.

Use the command `godot --gdscript-generate-tests
godot-source/modules/gdscript/tests/scripts` to update the .out files
with the current output (make sure the output are the expected values
before committing).

The tests themselves are part of the doctest suite so those can be
executed with `godot --test`.

Co-authored-by: Andrii Doroshenko (Xrayez) <xrayez@gmail.com>
Some tests need functions of ProjectSettings that rely on this being
available.
Since loading the config might use the resource path, it needs to be
set before that happens.
@vnen
Copy link
Member Author

vnen commented Apr 8, 2021

Another update to fix the CI. I didn't realize that it runs with a Mono build without the glue (we may want to revisit that if tests for Mono are introduced). Since I was initializing the languages to use GDScript, the absence of Mono glue made it crash. I changed to only initialize GDScript.

@akien-mga
Copy link
Member

I didn't realize that it runs with a Mono build without the glue (we may want to revisit that if tests for Mono are introduced).

On 3.x it generates the glue, but for master it cannot be done in CI with a non-headless build yet (but #47414 might solve that).

@Xrayez
Copy link
Contributor

Xrayez commented Apr 8, 2021

I took #47157 and created a new test script for it:

# missing-constant-string-for-assert.gd

func test():
    var got = "GoDot"
    var expected = "Godot"
    assert(got == expected, "Error, expected: " + expected)

Then generated corresponding out files:

godot --gdscript-generate-tests modules/gdscript/tests/scripts

The resulting file looks like this (with the same name):

GDTEST_ANALYZER_ERROR
Expected constant string for assert error message.

Then run the GDScript test suite specifically:

godot --test --test-suite="*GDScript*" --success
modules\gdscript\tests\gdscript_test_runner.cpp(180): SUCCESS: CHECK( result.passed ) is correct!
  values: CHECK( true )
  logged: "D:/src/godot/gd4/modules/gdscript/tests/scripts/parser-errors/missing-contant-string-for-assert.gd"
          ""

So I'd say it's fairly functional!

Not sure about the requirement of specifying the path to GDScript test scripts for --gdscript-generate-tests, I think it should default to "modules/gdscript/tests/scripts" (just like a recent change to --doctool).

For testers: don't forget to compile the engine with scons tests=yes.

@akien-mga akien-mga merged commit 200d9a7 into godotengine:master Apr 16, 2021
@akien-mga
Copy link
Member

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants