Failing Test: Lucene.Net.Index.TestDuelingCodecs::TestEquals()

[NightOwl888]

For information about how to help us debug tests, see #269.

This test has been failing randomly for some time but has not previously been documented. The test uses SimpleText to compare against all other codecs to ensure they return the same results.

Unfortunately the patch in #541 wasn't the only contributor to this failure.

I have confirmed there is a difference between Lucene46 and SimpleText, as it will fail when setting up the test as follows:

[assembly: Lucene.Net.Util.RandomSeed(0x4ba764c7d64c274dL)]
[assembly: NUnit.Framework.SetCulture("ts-ZA")]

In BeforeClass():

leftCodec = Codec.ForName("SimpleText");
rightCodec = Codec.ForName("Lucene46");

However, unlike the random failures below, this failure is repeatable when attaching a debugger.

Random Failures

Note that test failures are extremely rare. Doing 50 full runs (each run all of the tests 22 times on various platforms), there were only 2 failures.

It is 100% repeatable in .NET 5, x64, Windows with the following 2 seeds. While the test will fail with optimizations disabled, it doesn't fail when attaching a debugger. This makes it likely this has something to do with timing (i.e. thread safety).

Failure 1

Lucene.Net.Tests._E-I - netcoreapp3.1 - x86 - Linux | Tests failed: 1, passed: 540, ignored: 14TestEquals
Failed 8h ago on fv-az249-667
Duration0:00:19.900
Ownernot available
Date started11/24/2021, 11:14:52 AM
Date completed11/24/2021, 11:15:12 AM
Failing since7h ago
Failing since build4.8.0-ci0000002587

Expected: [61], Actual: [69]

left: SimpleText / right: Lucene46: {docid=PostingsFormat(name=MockSep), title=PostingsFormat(name=Direct), titleTokenized=PostingsFormat(name=Direct), body=Pulsing41 (freqCutoff=12, minBlockSize=85, maxBlockSize=253), date=PostingsFormat(name=MockSep)}, docValues:{sortedset=DocValuesFormat(name=Disk), sparsenumeric=DocValuesFormat(name=Disk), titleDV=DocValuesFormat(name=Lucene45)}

To reproduce this test result:

Option 1:

Apply the following assembly-level attributes:

[assembly: Lucene.Net.Util.RandomSeed(0x1811ee64ebda9716L)]
[assembly: NUnit.Framework.SetCulture("pa-Guru")]

Option 2:

Use the following .runsettings file:

<RunSettings>
  <TestRunParameters>
    <Parameter name="tests:seed" value="0x1811ee64ebda9716" />
    <Parameter name="tests:culture" value="pa-Guru" />
  </TestRunParameters>
</RunSettings>

See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
at Lucene.Net.Util.LuceneTestCase.AssertTermsEnumEquals(String info, IndexReader leftReader, TermsEnum leftTermsEnum, TermsEnum rightTermsEnum, Boolean deep) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2743
at Lucene.Net.Util.LuceneTestCase.AssertTermsEquals(String info, IndexReader leftReader, Terms leftTerms, Terms rightTerms, Boolean deep) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2664
at Lucene.Net.Util.LuceneTestCase.AssertFieldsEquals(String info, IndexReader leftReader, Fields leftFields, Fields rightFields, Boolean deep) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2618
at Lucene.Net.Util.LuceneTestCase.AssertReaderEquals(String info, IndexReader leftReader, IndexReader rightReader) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2581
at Lucene.Net.Index.TestDuelingCodecs.TestEquals() in D:\a\1\s\src\Lucene.Net.Tests\Index\TestDuelingCodecs.cs:line 178

See https://dev.azure.com/shad0962/Experiments/_build/results?buildId=1671&view=ms.vss-test-web.build-test-results-tab&runId=728436&resultId=100159&paneView=debug to download build artifacts for this run, but do note it will be cleaned up after 30 days.

Failure 2

Lucene.Net.Tests._E-I - netcoreapp3.1 - x86 - Windows | Tests failed: 1, passed: 527, ignored: 27TestEquals
Failed 8h ago on fv-az34-326
Duration0:00:02.620
Ownernot available
Date started11/24/2021, 12:07:47 PM
Date completed11/24/2021, 12:07:49 PM
Failing since6h ago
Failing since build4.8.0-ci0000001951

Expected: [61], Actual: [65]

left: SimpleText / right: Lucene46: {title=PostingsFormat(name=Memory doPackFST= True), docid=PostingsFormat(name=TestBloomFilteredLucene41Postings), body=PostingsFormat(name=TestBloomFilteredLucene41Postings), titleTokenized=PostingsFormat(name=TestBloomFilteredLucene41Postings), date=PostingsFormat(name=MockSep)}, docValues:{sparsenumeric=DocValuesFormat(name=Lucene45), sortedset=DocValuesFormat(name=Lucene45), titleDV=DocValuesFormat(name=Disk)}

To reproduce this test result:

Option 1:

Apply the following assembly-level attributes:

[assembly: Lucene.Net.Util.RandomSeed(0xbb8e501751632611L)]
[assembly: NUnit.Framework.SetCulture("en")]

Option 2:

Use the following .runsettings file:

<RunSettings>
  <TestRunParameters>
    <Parameter name="tests:seed" value="0xbb8e501751632611" />
    <Parameter name="tests:culture" value="en" />
  </TestRunParameters>
</RunSettings>

See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
at Lucene.Net.Util.LuceneTestCase.AssertTermsEnumEquals(String info, IndexReader leftReader, TermsEnum leftTermsEnum, TermsEnum rightTermsEnum, Boolean deep) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2743
at Lucene.Net.Util.LuceneTestCase.AssertTermsEquals(String info, IndexReader leftReader, Terms leftTerms, Terms rightTerms, Boolean deep) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2664
at Lucene.Net.Util.LuceneTestCase.AssertFieldsEquals(String info, IndexReader leftReader, Fields leftFields, Fields rightFields, Boolean deep) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2618
at Lucene.Net.Util.LuceneTestCase.AssertReaderEquals(String info, IndexReader leftReader, IndexReader rightReader) in D:\a\1\s\src\Lucene.Net.TestFramework\Util\LuceneTestCase.cs:line 2581
at Lucene.Net.Index.TestDuelingCodecs.TestEquals() in D:\a\1\s\src\Lucene.Net.Tests\Index\TestDuelingCodecs.cs:line 178

See https://dev.azure.com/lucene-net-temp2/Lucene.NET/_build/results?buildId=760&view=ms.vss-test-web.build-test-results-tab&runId=499080&resultId=100359&paneView=debug to download build artifacts for this run, but do note it will be cleaned up after 30 days.