Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Describe the process of writing integration tests for GDScript #4842

Merged
merged 1 commit into from
Jun 2, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions development/cpp/unit_testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -288,3 +288,62 @@ Here's an example of how GDScript registers test tools in

The custom command-line parsing can be performed by a test tool itself with the
help of OS :ref:`get_cmdline_args<class_OS_method_get_cmdline_args>` method.

Integration tests for GDScript
------------------------------

Godot uses doctest to prevent regressions in GDScript during development. There
are several types of test scripts which can be written:

- tests for expected errors;
- tests for warnings;
- tests for features.

Therefore, the process of writing integration tests for GDScript is the following:

1. Pick a type of a test script you'd like to write, and create a new GDScript
file under the ``modules/gdscript/tests/scripts`` directory within
corresponding sub-directory.

2. Write GDScript code. The test script must have a function called ``test()``
which takes no arguments. Such function will be called by the test runner.
The test should not have any dependency unless it's part of the test too.
Global classes (using ``class_name``) are registered before the runner
starts, so those should work if needed.

Here's an example test script:

::

func test():
if true # Missing colon here.
print("true")

3. Generate ``*.out`` files to update the expected results from the output:

.. code-block:: shell

./bin/<godot_binary> --gdscript-generate-tests godot-source/modules/gdscript/tests/scripts

4. Run GDScript tests with:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the not to check if the *.out files are expected should come before this step. If you generate the outputs and immediately run the test, it will always pass.

It should be clear also that if generating change other tests that you didn't add, you should revert it back and not commit.

Copy link
Contributor Author

@Xrayez Xrayez May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I understand correctly. Is it possible to run tests for new scripts that don't have *.out files generated for them yet? Not sure what would be the point of that, since there will be nothing to compare the output against.

I agree that it can be clarified that those generated out files which fail should not be committed, but a regression should reported (yet this likely won't be an issue, since regressions are going to be caught early at CI checks stages). This is also not a big deal for someone who participates in implementing GDScript language itself.

But current version does tell:

Make sure the output does have the expected values before submitting a pull request.

Copy link
Contributor Author

@Xrayez Xrayez May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is that, the process describes how to write new tests, not updating existing ones. If you believe it's beneficial to describe those processes separately, then it may be worth it. But I'm afraid this could potentially make the documentation more confusing.

Calinou went through those steps successfully while making godotengine/godot#48029, for instance. If someone is going to have a problem, the documentation could be further clarified I guess.

That said, not sure what kind of changes to make without breaking the currently described workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an explicit warning:

image


.. code-block:: shell

./bin/<godot_binary> --test --test-suite="*GDScript*"

If no errors are printed and everything goes well, you're done!

.. warning::

Make sure the output does have the expected values before submitting a pull
request. If ``--gdscript-generate-tests`` produces ``*.out`` files which are
unrelated to newly added tests, you should revert those files back and
only commit ``*.out`` files for new tests.

.. note::

The GDScript test runner is meant for testing the GDScript implementation,
not for testing user scripts nor testing the engine using scripts. We
recommend writing new tests for already resolved
`issues related to GDScript at GitHub <https://github.com/godotengine/godot/issues?q=is%3Aissue+label%3Atopic%3Agdscript+is%3Aclosed>`_,
or writing tests for currently working features.