[JENKINS-69650] Fix test reporting of Security2779Test

[MarkEWaite]

JENKINS-69650 Fix test reporting of Security2779Test

Apache Maven surefire plugin does not seem to fully support parameterized JUnit 4 tests. See junit-team/junit5#990 for discussion of the issue.

The surefire plugin acts as though the Security2779Test is being run multiple times with the same argument and treats a failure of one argument as a "flaky test". In this case, the test is not flaky as far as I can tell, it is showing a real failure when the argument is "#icon-panel svg".

The test passes consistently when the argument is "#link-panel a".

The consistent failures that I've seen were on Red Hat Enterprise Linux 8.6 with Eclipse Temurin Java 11.0.16.1. The same failure was seen in the Jenkins 2.369 weekly build.

This commit does not fix the failure, it just makes it easier to see that there is a failure.

It appears that the parameterized test started failing for the '#icon-panel svg' with commit 3923e45 in pull request #7052. See the bisection command used in JENKINS-69650

Proposed changelog entries

• No changelog - test related only

Proposed upgrade guidelines

N/A

Submitter checklist

• (If applicable) Jira issue is well described
• Changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developer, depending on the change) and are in the imperative mood. Examples
  • Fill-in the Proposed changelog entries section only if there are breaking changes or other changes which may require extra steps from users during the upgrade
• Appropriate autotests or explanation to why this change has no tests
• New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadoc, as appropriate.
• New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO") if applicable.
• New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content-Security-Policy directives (see documentation on jenkins.io).
• For dependency updates: links to external changelogs and, if possible, full diffs

Desired reviewers

@NotMyFault, @daniel-beck

Maintainer checklist

Before the changes are marked as ready-for-merge:

• There are at least 2 approvals for the pull request and no outstanding requests for change
• Conversations in the pull request are over OR it is explicit that a reviewer does not block the change
• Changelog entries in the PR title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood
• Proper changelog labels are set so that the changelog can be generated automatically
• If the change needs additional upgrade steps from users, upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the PR title. (example)
• If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

[jglick]

@Parameterized is often a mess. Simpler & clearer (IMO) to get rid of it and just have one test method run assertions on both selectors. Or define two @Test methods which simply delegate to a private implementation.

[jglick]

Thanks!

[daniel-beck]

While 84f41d2 in 2.370 addresses the failure, this change is still worthwhile.

[jglick]

The surefire plugin acts as though the Security2779Test is being run multiple times with the same argument and treats a failure of one argument as a "flaky test".

This sounds like SUREFIRE-2087 though that is reported for JUnit 5. I wonder if the same applies to JUnit 4? Or we caused a 5-specific bug to start affecting our codebase by introducing the 5 libraries with the vintage engine? It would be nice if someone could reproduce this in a self-contained test case using 4 (with or without 5 in the classpath) and comment accordingly in SUREFIRE-2087.

[MarkEWaite]

@daniel-beck and @NotMyFault are there additional changes needed for this pull request before it is approved and starts the 24 hour countdown timer to merge?

[NotMyFault]

Looks good, no review or merge blocker from my side.

Thanks for taking care of it :)

[timja]

/label ready-for-merge

---

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!