-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PluginVal shows "ALL TESTS PASSED" even when completely failed at "Editor" stage #125
Comments
How are you running pluginval? It's not quite clear from the screenshot. Thanks. |
In the OP screenshot I ran PluginVal in the "Test File" mode with the settings shown below - i.e., one specific plugin at a time. Previously I ran it on a batch of plugins and that is when I discovered this issue. If you're not paying attention you can think "ALL TESTS PASSED" means everything is good but that's not always true as I show here. |
Any idea how I can replicate this? I'll be honest, it's been a while since I've been in this code and a quick repo case will help me much more quickly identify the issue. Thanks in advance. |
I am trying to remember in this particular case what the problem turned out to be. I was anxious to get it fixed ASAP so I didn't think much about keeping a backup of the errant code. Segmentation Fault error 11 is usually about memory issues. I think in this case I had a "picket fence" error in an array. Whatever the cause, Pluginval should have stopped when any part of the test failed and report FAILED at the end rather than showing ALL TESTS PASSED. |
Yeah, I agree, it's just that it should have intercepted the segfault (SIGSEGV`) as can be seen here: pluginval/Source/CommandLine.cpp Line 70 in 0f8e91b
I need to know in what situation the problem arises before I can fix it. |
I think you just showed the situation that needs fixing - when ANY PART of the testing fails the end report should say "FAILED". Now as to why it failed at the Editor stage could be any number of things. If I can find a backup version of the plugin that did this I will post it. In the meantime maybe another user can provide some input. |
Ok, I think I've got a repo for this now. It only seems to happen when you have multiple plugins being tested. |
FYI I think this is a juce issue I've raised here: https://forum.juce.com/t/killed-childprocess-activeprocess-exit-code/61645/3 |
In the example I posted here it did this when testing only 1 plugin at a time. Please see my screenshots above. |
Came across a very misleading situation where PluginVal reports "ALL TESTS PASSED" when in fact it failed miserably at the loading Editor stage due to graphics issues, etc.
If one is not paying attention - and carefully reading the report - it can appear that the plugin is GOOD when in fact it is a complete failure. Please see example here:
The text was updated successfully, but these errors were encountered: