-
-
Notifications
You must be signed in to change notification settings - Fork 21.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Doctest regression suite for gdscript, loading scripts and corresponding par… #41616
Conversation
958e403
to
db50048
Compare
Basically @vnen already created GDScript test runner, what I've done is to adapt his existing system to doctest. I stopped working on #41074 as I waited for Godot to merge #40980, but now it seems to be functional. Also, @vnen doesn't seem to be enthusiastic regarding adding tests to the main repo, so I'm not sure. I personally think including tests in the main repo makes sense, especially since tests can be implemented per-module #40720, btw.
That's strange, make sure to include double quotes. |
Cheers @Xrayez ! I figured there must be something already - writing a full tokenizer/parser/compiler without a test-suite would have been... impressive ;) |
d864647
to
0ad25f1
Compare
0ad25f1
to
6450e1a
Compare
The tests should likely be moved to
I also wonder about this. If we want to have a full test suite for all GDScript features, each time with a |
…enized files from a directory. Automatically tests all .gd files, intended as the start of a regression suite. Could be improved by: * avoiding code duplication with gdscript test command, see note in test_gdscript_doctest.h * showing a diff on failure, rather than the default doctest output of showing both sides of == below each other. Easiest way is to generate the diff, somehow, and then compare against the empty output.
6450e1a
to
cb7f4fd
Compare
The results of the tokenizer/parser output aren't good for regression testing. Fixing a bug might change their output so there would be many cases of false positives whenever we add a small fix. The test suite I proposed, which @Xrayez adapted to doctest in #41074, uses more high-level information such as error messages and console output. This way we can test the behavior of the script without caring about how the compiler is implemented. If we had this before my rewrite would be much simpler to test, while using the parse tree output would be useless since the tree structure changed a lot. The tree printer is only useful for debugging, so we can manually check what went wrong. |
I agree that during a complete re-write, only the highest level tests are useful. I still think that minimal test cases for the tokenizer and parser are useful in a normal regression suite - minimal in the sense that they are as small as possible, so unrelated small fixes do not lead to too much work in updating the tests. I'll set this one back to draft (it was just an offshoot of trying for a not-in operator) and see if I can help with #41074. |
If I understand correctly, this could be implemented using doctest decorators for test cases: |
Hmm. I'm not sure if |
@vnen Can you give a status update on this? |
Reiterating my comment, the output of those tests are not standard (and also not fully validated). Those are useful for debugging, but fixing actual bugs will likely change the output as well, which also makes it not unfeasible that the test misses a regression. Since #47701 is now merged, we should use that system to test the issues and catch the regressions, since it's much less dependent on implementation details. |
…se trees from a directory.
This relates to godotengine/godot-proposals#1319 but might be interesting in itself, for @vnen I assume.
Should be easy to add more test cases, or integrate the ones already in place somewhere outside the repo.
Use
./bin/godot --test -tc="*GDScript*"
to run the one test only. For some reason the long-form option --test-case=... does not work.