System.Text.RegularExpressions.Tests.AttRegexTests test fails in CI

[jkotas]

Runfo failures in last 30 days

As of 9/24:

Day	Run	OS	Details
9/24	Rolling run 28605	net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open	Console log
~9/18	PR Jan's original report		Not listed in Runfo / Kusto due to infra hiccups

No failures in Kusto as of 9/24

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    '', //testNameSubstring
    'AttRegexTests', //methodName
    '', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

Console log on 9/24 (rolling run 28605):

    System.Text.RegularExpressions.Tests.AttRegexTests.Test(engine: Compiled, options: Multiline, pattern: "a[bcd]*dcdcde", input: "adcdcde", expected: "(0,7)", skipNonBacktracking: False) [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/AttRegexTests.cs(409,0): at System.Text.RegularExpressions.Tests.AttRegexTests.Test(RegexEngine engine, RegexOptions options, String pattern, String input, String expected, Boolean skipNonBacktracking)

Original report

  Starting:    System.Text.RegularExpressions.Tests (parallel test collections = on, max threads = 6)
    System.Text.RegularExpressions.Tests.AttRegexTests.Test(engine: Compiled, options: None, pattern: ":::1:::0:|:::1:1:0:", input: ":::0:::1:::1:::0:", expected: "(8,17)", skipNonBacktracking: False) [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/AttRegexTes

Hit in #75805. Log https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75805-merge-b1ca28fe798046d3af/System.Text.RegularExpressions.Tests/1/console.2a2dc4a5.log?helixlogtype=result

{
    "ErrorMessage" : "System.Text.RegularExpressions.Tests.AttRegexTests.Test",
     "BuildRetry": false
}

Report

Build	Definition	Test	Pull Request
338141	dotnet/runtime	System.Text.RegularExpressions.Tests.RegexMatchTests.Match

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

  Starting:    System.Text.RegularExpressions.Tests (parallel test collections = on, max threads = 6)
    System.Text.RegularExpressions.Tests.AttRegexTests.Test(engine: Compiled, options: None, pattern: ":::1:::0:|:::1:1:0:", input: ":::0:::1:::1:::0:", expected: "(8,17)", skipNonBacktracking: False) [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/AttRegexTes

Hit in #75805. Log https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75805-merge-b1ca28fe798046d3af/System.Text.RegularExpressions.Tests/1/console.2a2dc4a5.log?helixlogtype=result

Author:	jkotas
Assignees:	-
Labels:	area-System.Text.RegularExpressions, untriaged
Milestone:	-

[joperezr]

cc: @stephentoub

This is very weird; those tests are supposed to be deterministic. I understand that this is a theory that in reallity involves hundreds of tests due to all of the permutations we make, so grepping for failures of the test doesn't really equate to having a Fact that fails as often, but I still don't understand why any of these tests would ever fail indeterministic, the line that is failing is literally just evaluating if a given input matches a given Regex object. Any clues why this may be flaky Steve?

Not sure if it is relevant, but in the query that @jkotas provided above, this has failed 4 times in rolling in the last 30 days, and it has always been in the queue net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open

[jkotas]

It is likely a GC hole. Does this test calls into the recently vectorized methods? They would be my primary suspect.

Here are a few things you can do to help diagnose it:

• Make the test to fail-fast (call Environment.FailFast on mismatch) so that we get crash dump close to the failure. Run the test locally with stresslog enabled (Process crash with "Internal CLR error" (0x80131506) in Roslyn CI #45557 (comment)) until you get a crash dump.
• Try to reduce the repro. For example, extract the test and run it in a loop on multiple thread to see whether you are able to get it to crash.

[stephentoub]

Does this test calls into the recently vectorized methods?

The particular pattern shown in the opening issue comment will end up using ReadOnlySpan<char>.IndexOf(ReadOnlySpan<char>) and ReadOnlySpan<char>.StartsWith(ReadOnlySpan<char>).

[jkotas]

Similar failures in a different test https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75612-merge-564fd52802984edbad/System.Text.RegularExpressions.Tests/1/console.09dea590.log?helixlogtype=result :

  Starting:    System.Text.RegularExpressions.Tests (parallel test collections = on, max threads = 2)
    System.Text.RegularExpressions.Tests.RegexGroupTests.Groups(engine: Compiled, cultureName: null, pattern: "(cat)(\\176)", input: "hellocatcat76dogworld", options: ECMAScript, expectedGroups: ["catcat76", "cat", "cat76"]) [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:

[stephentoub]

That compiled regex would also end up using ReadOnlySpan<char>.IndexOf(ReadOnlySpan<char>).

[steveharter]

I am going to close the issue since no activity here or for "AttRegexTests" in 10 months, please reopen it if it fails again - at that point we can fill out the "known issue" template to track the failures.

The reference above to System.Text.RegularExpressions.Tests.RegexGroupTests.Groups is discussed in #65353 which was fixed.