Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PluginVal shows "ALL TESTS PASSED" even when completely failed at "Editor" stage #125

Closed
rexmartin158 opened this issue May 17, 2024 · 9 comments

Comments

@rexmartin158
Copy link

rexmartin158 commented May 17, 2024

Came across a very misleading situation where PluginVal reports "ALL TESTS PASSED" when in fact it failed miserably at the loading Editor stage due to graphics issues, etc.

If one is not paying attention - and carefully reading the report - it can appear that the plugin is GOOD when in fact it is a complete failure. Please see example here:

PluginVal failed test

@drowaudio
Copy link
Contributor

How are you running pluginval? It's not quite clear from the screenshot.

Thanks.

@rexmartin158
Copy link
Author

How are you running pluginval? It's not quite clear from the screenshot.

Thanks.

In the OP screenshot I ran PluginVal in the "Test File" mode with the settings shown below - i.e., one specific plugin at a time. Previously I ran it on a batch of plugins and that is when I discovered this issue. If you're not paying attention you can think "ALL TESTS PASSED" means everything is good but that's not always true as I show here.

PluginVal failed test mode

@drowaudio
Copy link
Contributor

Any idea how I can replicate this? I'll be honest, it's been a while since I've been in this code and a quick repo case will help me much more quickly identify the issue. Thanks in advance.

@rexmartin158
Copy link
Author

Any idea how I can replicate this? I'll be honest, it's been a while since I've been in this code and a quick repo case will help me much more quickly identify the issue. Thanks in advance.

I am trying to remember in this particular case what the problem turned out to be. I was anxious to get it fixed ASAP so I didn't think much about keeping a backup of the errant code. Segmentation Fault error 11 is usually about memory issues. I think in this case I had a "picket fence" error in an array.

Whatever the cause, Pluginval should have stopped when any part of the test failed and report FAILED at the end rather than showing ALL TESTS PASSED.

@drowaudio
Copy link
Contributor

Yeah, I agree, it's just that it should have intercepted the segfault (SIGSEGV`) as can be seen here:

const int signals[] = { SIGFPE, SIGILL, SIGSEGV, SIGBUS, SIGABRT };

I need to know in what situation the problem arises before I can fix it.

@rexmartin158
Copy link
Author

Yeah, I agree, it's just that it should have intercepted the segfault (SIGSEGV`) as can be seen here:

const int signals[] = { SIGFPE, SIGILL, SIGSEGV, SIGBUS, SIGABRT };

I need to know in what situation the problem arises before I can fix it.

I think you just showed the situation that needs fixing - when ANY PART of the testing fails the end report should say "FAILED".

Now as to why it failed at the Editor stage could be any number of things. If I can find a backup version of the plugin that did this I will post it. In the meantime maybe another user can provide some input.

@drowaudio
Copy link
Contributor

Ok, I think I've got a repo for this now. It only seems to happen when you have multiple plugins being tested.
I'm guessing it doesn't propagate the exit code on correctly. Will have a look.

@drowaudio
Copy link
Contributor

FYI I think this is a juce issue I've raised here: https://forum.juce.com/t/killed-childprocess-activeprocess-exit-code/61645/3

@rexmartin158
Copy link
Author

Ok, I think I've got a repo for this now. It only seems to happen when you have multiple plugins being tested.

In the example I posted here it did this when testing only 1 plugin at a time. Please see my screenshots above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants