Integrate GDScript tests to use doctest

[Xrayez]

Test data moved from: https://github.com/godotengine/gdscript-tests

Currently this PR makes it possible to run those script test data against latest master branch with the exact command-line tool as in vnen@1db7832.

You can still use --gdscript-generate-tests to manually generate/update expected output for testing.

You can run GDScript-only tests with this command for testing purposes:

godot --test -sf="*test_gdscript*" -s

You'll have to compile with scons tests=yes.

[Xrayez]

Static checks fail on GDScript and test data... 🤔

[ThakeeNathees]

all files should end with a new line i guess

*.out files could be gitignored ?

[Xrayez]

*.out files could be gitignored ?

Those files are needed to compare generated results from scripts to expected results AFAIK, I haven't delved into this yet. And yeah it would rely on error messages I suppose? The *.out files would have to be manually updated I think.

[Xrayez]

Managed to run those tests through doctest now!

Current output:

[doctest] doctest version is "2.4.0"
[doctest] run with "--help" for options
ERROR: %c requires number or single-character string
   at: (core\variant.cpp:3518)
ERROR: %c requires number or single-character string
   at: (core\variant.cpp:3518)
===============================================================================
D:\src\godot\modules\gdscript\tests\test_gdscript.h(43):
TEST SUITE: [Modules][GDScript]
TEST CASE:  Script loading, parsing and compiling

modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/missing-argument.gd
          GDTEST_PARSER_ERROR
Expression expected


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/missing-closing-expr-paren.out
          GDTEST_PARSER_ERROR
Expected ')' in expression


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/missing-colon.gd
          GDTEST_PARSER_ERROR
':' expected at end of line.


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/missing-paren-after-args.gd
          GDTEST_PARSER_ERROR
Expected ',' or ')'


ERROR: %c requires number or single-character string
   at: (core\variant.cpp:3518)
modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/mixing-tabs-spaces.gd
          GDTEST_PARSER_ERROR
Mixed tabs and spaces in indentation.


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/nothing-after-dollar.gd
          GDTEST_PARSER_ERROR
Path expected after $.


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/wrong-value-after-dollar-slash.gd
          GDTEST_PARSER_ERROR
Expected string constant or identifier after '$' or '/'.


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-errors/wrong-value-after-dollar.gd
          GDTEST_PARSER_ERROR
Expected string constant or identifier after '$' or '/'.


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-features/semicolon-as-end-statement.gd
          GDTEST_OK
A
B


ERROR: %c requires number or single-character string
   at: (core\variant.cpp:3518)
modules\gdscript\tests\gdscript_test.cpp(84): SUCCESS: CHECK( result.passed ) is correct!
  values: CHECK( true )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-features/trailing-comma-in-function-args.gd
          GDTEST_PARSER_ERROR


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-features/variable-declaration.gd
          GDTEST_OK
1 42 2 23
OK


modules\gdscript\tests\gdscript_test.cpp(84): ERROR: CHECK( result.passed ) is NOT correct!
  values: CHECK( false )
  logged: D:/src/godot/modules/gdscript/tests/scripts/parser-warnings/unused-variable.out
          GDTEST_OK
>> WARNING
>> Line: 2
>> The local variable 'unused' is declared but never used in the block. If this is intended, prefix it with an underscore: '_unused'


D:\src\godot\modules\gdscript\tests\test_gdscript.h(48): FATAL ERROR: REQUIRE( fail_count == 0 ) is NOT correct!
  values: REQUIRE( 11 == 0 )
  logged: Make sure `*.out` files have expected results.
          All GDScript tests should pass.

Allowed to fail so marking it as not failed
===============================================================================
[doctest] test cases:      1 |      1 passed |      0 failed |     50 skipped
[doctest] assertions:     13 |      1 passed |     12 failed |
[doctest] Status: SUCCESS!

Most fail because I haven't updated the *.out files (and I have no clue what's the expected behavior should be), and current implementation make those tests fail currently. But I added a single successful case for recently confirmed feature which is not a bug: #41066, so that is marked as SUCCESS there (interestingly, it still reports GDTEST_PARSER_ERROR, but the got == expected works, so we've got a half-feature 😛).

CI fails because Godot needs fixes from #40980, and then likely have to come up with some test contexts godotengine/godot-proposals#1307, so I have to suspend the work for now. I might be able to fix this for current PR locally, but I'd prefer if we review and discuss previous PRs and proposals first, I think I can fix those issues eventually.

[Xrayez]

I currently find it problematic to work on this given #41307 on Windows...

See also #41616.

Created proposal: godotengine/godot-proposals#1429.

[Xrayez]

I guess the direction of godotengine/godot-proposals#1429 is that we add test data as a submodule for Godot... I think I could try this out and see the feasibility/practicality of this.

[Xrayez]

I'm not quite sure why it fails on Linux/Windows after rebase for now, but this is again testable.

I'm not adding scripts via submodule at the moment, godotengine/godot-proposals#1429, should simplify the work on the PR for now, not high priority.

Things to do:

• Can likely port --gdscript-generate-tests path/to/scripts and --gdscript-test script.gd to use test commands as in Port GDScript test/debugging tools #41355.
• Properly identify the got/expected results to make it unambiguous, especially when testing for failure paths.

[Xrayez]

Forgot to mention that the expected output could be potentially configured with a custom reporter: #40890, I see this could be more like diff output, but not sure if trivial to implement yet.

[vnen]

@Xrayez I've poked into this today and I believe I solved all the issues. I've also submitted #47654 to suppress the output from scripts (which could potentially be used in other tests). Not sure if you want to keep supporting this PR (so I add my fixes on top of your branch) or if I should open a new one.

Regarding some extra changes (custom reporting, moving the generate command to test commands) those could be done later as enhancements. We could also review the output format if that makes sense (right now it's just the expected error in the first line, followed by warnings (if there wasn't any parser error), followed by either an error message or the script output). I've also changed to only use the first error message since the following ones are likely cascading from the first and are subject to change (and not really relevant for the tests themselves).

[Xrayez]

@vnen feel free to open a new PR to supersede this one.

[akien-mga]

Superseded by #47701.