corruption_test failed: db/corruption_test.cc:325: failed: 1 == 5

[cmumford]

Original issue 87 created by tfengjun on 2012-05-07T06:21:36.000Z:

What steps will reproduce the problem?

1. make corruption_test
2. for((i=1;;i++)); do
   ./corruption_test
   if [[ $? -ne 0 ]]; then
   break;
   fi
   done
   3.

What is the expected output? What do you see instead?

Expected output:
==== Test CorruptionTest.Recovery
expected=100..100; got=100; bad_keys=0; bad_values=0; missed=0
expected=36..36; got=36; bad_keys=0; bad_values=0; missed=64
==== Test CorruptionTest.RecoverWriteError
==== Test CorruptionTest.NewFileErrorDuringWrite
==== Test CorruptionTest.TableFile
expected=99..99; got=99; bad_keys=0; bad_values=1; missed=0
==== Test CorruptionTest.TableFileIndexData
expected=5000..9999; got=7958; bad_keys=0; bad_values=0; missed=0
==== Test CorruptionTest.MissingDescriptor
expected=1000..1000; got=1000; bad_keys=0; bad_values=0; missed=0
==== Test CorruptionTest.SequenceNumberRecovery
==== Test CorruptionTest.CorruptedDescriptor
==== Test CorruptionTest.CompactionInputError
expected=9..9; got=9; bad_keys=0; bad_values=1; missed=0
expected=10000..10000; got=10000; bad_keys=0; bad_values=0; missed=0
==== Test CorruptionTest.CompactionInputErrorParanoid
expected=9..9; got=9; bad_keys=2; bad_values=1; missed=0
==== Test CorruptionTest.UnrelatedKeys
==== PASSED 11 tests

What I see:
==== Test CorruptionTest.Recovery
expected=100..100; got=100; bad_keys=0; bad_values=0; missed=0
expected=36..36; got=36; bad_keys=0; bad_values=0; missed=64
==== Test CorruptionTest.RecoverWriteError
==== Test CorruptionTest.NewFileErrorDuringWrite
==== Test CorruptionTest.TableFile
expected=99..99; got=99; bad_keys=0; bad_values=1; missed=0
==== Test CorruptionTest.TableFileIndexData
expected=5000..9999; got=7958; bad_keys=0; bad_values=0; missed=0
==== Test CorruptionTest.MissingDescriptor
expected=1000..1000; got=1000; bad_keys=0; bad_values=0; missed=0
==== Test CorruptionTest.SequenceNumberRecovery
==== Test CorruptionTest.CorruptedDescriptor
==== Test CorruptionTest.CompactionInputError
expected=9..9; got=9; bad_keys=0; bad_values=1; missed=0
expected=10000..10000; got=10000; bad_keys=0; bad_values=0; missed=0
==== Test CorruptionTest.CompactionInputErrorParanoid
db/corruption_test.cc:325: failed: 1 == 5

What version of the product are you using? On what operating system?
1.4, git commit "85584d497e7b354853b72f450683d59fcf6b9c5c"
OS: CentOS 5.4, kernel 2.6.35.13, x64 virtual machine(VMWare)

Please provide any additional information below.

[cmumford]

Comment #1 originally posted by tfengjun on 2012-05-07T06:40:51.000Z:

You need to run the for loop for serveral minutes before you see the failed result.
If you're lucky enough, you can see it hanppens very soon(it once failed in the second run).

[cmumford]

Comment #2 originally posted by tfengjun on 2012-05-07T06:41:39.000Z:

You need to run the for loop for serveral minutes before you see the failed result.
If you're lucky enough, you can see it happens very soon(it once failed in the second run).

[cmumford]

Comment #3 originally posted by tfengjun on 2012-05-08T02:07:56.000Z:

I was not able to reproduce the bug in a real linux box, but it happened on 3 vm instances runing CentOS 5.4, with different kernels including: 2.6.18-238, 2.6.18-164, 2.6.35.8 and 2.6.35.13.

Maybe it's a vm issue??

[cmumford]

Comment #4 originally posted by der.pitti on 2013-06-04T15:32:01.000Z:

In Ubuntu this happens on real hardware without VMs, but only on ARM:
https://launchpad.net/ubuntu/+source/leveldb/1.10.0-1/+build/4616412
https://launchpad.net/ubuntu/+source/leveldb/1.10.0-1git1/+build/4640626

What does the "leveldb.num-files-at-level0" property mean, anyway? Is that test maybe too strict, and some ">= 1" might be more appropriate?

[cmumford]

Comment #5 originally posted by der.pitti on 2013-06-04T15:52:54.000Z:

Note, I re-tried the second build (https://launchpad.net/ubuntu/+source/leveldb/1.10.0-1git1/+build/4640626) and that worked the second time, so this confirms the race condition from the initial description, and that it is not related to a VM.

[cmumford]

Comment #6 originally posted by dgrogan@chromium.org on 2013-06-05T16:51:25.000Z:

der.pitti, I don't see the CorruptionTest.Recovery issue at all in the links you gave, just an internal compiler error in the first one. Am I missing something?

[cmumford]

Comment #7 originally posted by der.pitti on 2013-06-06T08:05:12.000Z:

Indeed, I'm sorry. It seems that only happened in the 1git1 build, and as I retried the build until that "1 == 5" failure went away. It looked exactly like the one in the description.

[cmumford]

Comment #8 originally posted by fullung on 2013-06-28T04:04:59.000Z:

Apparently this is due to a race in test:

https://code.google.com/p/leveldb/issues/detail?id=182#c3

[cmumford]

Comment #9 originally posted by sanjay@google.com on 2013-07-01T21:31:34.000Z:

I have been able to reproduce this problem and I am working on a fix. Thanks for the reports.

[cmumford]

Comment #10 originally posted by dgrogan@chromium.org on 2013-12-10T19:51:34.000Z:

Fixed in 1.13