PerformanceCounter tests failing in CI

[stephentoub]

Inconsistently but frequently, PRs in CI have been failing tests from the System.Diagnostics.Tests.PerformanceCounterTests suite. When one test fails, a ton fail. They all look along the lines of this:

Error message
System.InvalidCastException : Unable to cast object of type 'System.String' to type 'System.String[]'.

Stack trace
   at System.Diagnostics.SharedPerformanceCounter.GetCategoryData() in /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/SharedPerformanceCounter.cs:line 700
   at System.Diagnostics.SharedPerformanceCounter..ctor(String catName, String counterName, String instanceName, PerformanceCounterInstanceLifetime lifetime) in /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/SharedPerformanceCounter.cs:line 96
   at System.Diagnostics.SharedPerformanceCounter..ctor(String catName, String counterName, String instanceName) in /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/SharedPerformanceCounter.cs:line 88
   at System.Diagnostics.SharedPerformanceCounter.RemoveAllInstances(String categoryName) in /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/SharedPerformanceCounter.cs:line 1389
   at System.Diagnostics.PerformanceCounterCategory.Delete(String categoryName) in /_/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/PerformanceCounterCategory.cs:line 395
   at System.Diagnostics.Tests.Helpers.DeleteCategory(String name) in /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs:line 46
   at System.Diagnostics.Tests.Helpers.CreateCategory(String name, PerformanceCounterCategoryType categoryType) in /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/Helpers.cs:line 22
   at System.Diagnostics.Tests.PerformanceCounterTests.CreateCounterWithCategory(String name, Boolean readOnly, PerformanceCounterCategoryType categoryType) in /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs:line 293
   at System.Diagnostics.Tests.PerformanceCounterTests.PerformanceCounter_Decrement() in /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/PerformanceCounterTests.cs:line 186

e.g. https://dev.azure.com/dnceng/public/_build/results?buildId=874236&view=ms.vss-test-web.build-test-results-tab&runId=27965576&resultId=121120&paneView=debug

[ghost]

Tagging subscribers to this area: @tommcdon, @krwq
See info in area-owners.md if you want to be subscribed.

[danmoseley]

Apparently "Counter Names" may be String or Multistring? the code expects the latter.

My guess is that here

runtime/src/libraries/System.Diagnostics.PerformanceCounter/src/System/Diagnostics/SharedPerformanceCounter.cs

Lines 700 to 703 in c87e75e

	string[] counterNames = (string[])counterNamesObject;
	for (int i = 0; i < counterNames.Length; i++)
	counterNames[i] = counterNames[i].ToLowerInvariant();
	data.CounterNames = new ArrayList(counterNames);

It simply needs to use GetValueKind to figure whether to cast to string or string[] and then proceed.

[danmoseley]

@Anipik owns S.D.PerformanceCounter.

[stephentoub]

@steveisok hit the new assert:

Process terminated. Assertion failed.
Expected string[], got 'System.String[]' with kind 'String'for category 'averagecounter64samplecategory'
   at System.Diagnostics.Tests.CounterCreationDataCollectionTests.PerformanceCounterCategory_CreateCategory() in /_/src/libraries/System.Diagnostics.PerformanceCounter/tests/CounterCreationDataCollectionTests.cs:line 124

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-44046-merge-146881d67d7c4c7ba9/System.Diagnostics.PerformanceCounter.Tests/console.210349c5.log?sv=2019-07-07&se=2020-11-25T20%3A35%3A23Z&sr=c&sp=rl&sig=gYGgvImTcGJ99mVVeSk%2Bg%2BucMp92%2F6xeEerbQTpkqv8%3D

#44332

[stephentoub]

This assert output is really weird. The assert I added is:

Debug.Assert(counterNamesObject is string[], $"Expected string[], got '{counterNamesObject}' with kind '{categoryKey.GetValueKind("Counter Names")}'for category '{_categoryName}'");

So the above shows it failing an is string[] check, but then its ToString is reporting that it's a System.String[].

@jkotas, does this ring any bells? Could this be related to casting changes?

Unless... maybe it actually is a String with the value "System.String[]"?! I'll update the assert.

[danmoseley]

Well, that's our test

runtime/src/libraries/System.Diagnostics.PerformanceCounter/tests/CounterCreationDataCollectionTests.cs

Line 99 in 6072e4d

public static void PerformanceCounterCategory_CreateCategory()

But it's just going through our API's.

[danmoseley]

Oops, of course it's written by our test 😄

[steveisok]

I tried finding this kind of issue first. I searched for "PerformanceCounters" and found 3 unrelated issues :-)

[jkotas]

@jkotas, does this ring any bells? Could this be related to casting changes?

Yes, I think this looks like race condition in the lock-free hashtable used by casting.

@VSadov Could you please take a look?

[VSadov]

Will take a look. Thanks!

[VSadov]

@stephentoub - did this start recently or was like this for months?

[stephentoub]

To my knowledge, recently.

[VSadov]

Exception message should generally be correct about types.
string[] --> string[] would be an identity conversion. It should not use cache, unless something bad happened with the type system (i.e. two distinct string types)

I think it is more likely that we are getting a string with System.String[] content somehow.

The new assert in #44333 is a good idea - should help to verify this.

[VSadov]

Any new failures? Is there a way to check?

[stephentoub]

I haven't seen any more but I'm also not at my computer to run a query right now. (I'm also wondering if this could possibly have been another effect of #44250? I haven't lined things up exactly, but the timing seems to correlate.)

[stephentoub]

According to Kusto, this failed ~300 times on 11/2, ~100 times on 11/3, and then hasn't failed since.

[jkotas]

Likely another effect of #44250