Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

windows: test flakes / failures due to out of memory #1897

Closed
bananabrick opened this issue Aug 16, 2022 · 7 comments
Closed

windows: test flakes / failures due to out of memory #1897

bananabrick opened this issue Aug 16, 2022 · 7 comments
Assignees
@nicktrav
Labels
A-storage T-storage

Comments

@bananabrick
Copy link
Contributor

https://github.com/cockroachdb/pebble/runs/7867395379?check_suite_focus=true

=== RUN   TestIndexedBatchMutation

testdata/indexed_batch_mutation:2:
new-batch [0 args]
set foo foo
----

testdata/indexed_batch_mutation:8:
new-batch-iter [1 args]
<no input to command>

Not sure what to make of the test output. Seems like a flake because it passed the second time around. Worth a look.
@jbowens
Copy link
Collaborator

jbowens commented Aug 16, 2022

Looks like it was actually TestDiskHealthChecking_Filesystem that flaked (#1718). The github actions log UI is horrendously slow, so I think you were just seeing a section of the log that had buffered so far. If you click on the gear icon and select 'View raw logs', it's usually easier to find the failure.

2022-08-16T21:29:46.3156052Z === RUN   TestDiskHealthChecking_Filesystem
2022-08-16T21:29:46.3156234Z === RUN   TestDiskHealthChecking_Filesystem/remove
2022-08-16T21:29:46.3156487Z === RUN   TestDiskHealthChecking_Filesystem/remove-all
2022-08-16T21:29:46.3156672Z === RUN   TestDiskHealthChecking_Filesystem/rename
2022-08-16T21:29:46.3156958Z === RUN   TestDiskHealthChecking_Filesystem/reuse-for-write
2022-08-16T21:29:46.3157158Z     assertion_compare.go:313: 
2022-08-16T21:29:46.3157453Z         	Error Trace:	disk_health_test.go:276
2022-08-16T21:29:46.3157730Z         	Error:      	"0" is not greater than "0"
2022-08-16T21:29:46.3158399Z         	Test:       	TestDiskHealthChecking_Filesystem/reuse-for-write
2022-08-16T21:29:46.3159047Z         	Messages:   	[]
2022-08-16T21:29:46.3159189Z === RUN   TestDiskHealthChecking_Filesystem/create
2022-08-16T21:29:46.3159341Z === RUN   TestDiskHealthChecking_Filesystem/link
2022-08-16T21:29:46.3159572Z === RUN   TestDiskHealthChecking_Filesystem/mkdir-all
2022-08-16T21:29:46.3159800Z --- FAIL: TestDiskHealthChecking_Filesystem (0.50s)
2022-08-16T21:29:46.3160451Z     --- PASS: TestDiskHealthChecking_Filesystem/remove (0.06s)
2022-08-16T21:29:46.3160977Z     --- PASS: TestDiskHealthChecking_Filesystem/remove-all (0.06s)
2022-08-16T21:29:46.3161303Z     --- PASS: TestDiskHealthChecking_Filesystem/rename (0.06s)
2022-08-16T21:29:46.3162193Z     --- FAIL: TestDiskHealthChecking_Filesystem/reuse-for-write (0.12s)
2022-08-16T21:29:46.3162885Z     --- PASS: TestDiskHealthChecking_Filesystem/create (0.06s)
2022-08-16T21:29:46.3163231Z     --- PASS: TestDiskHealthChecking_Filesystem/link (0.06s)
2022-08-16T21:29:46.3163598Z     --- PASS: TestDiskHealthChecking_Filesystem/mkdir-all (0.06s)

@bananabrick
Copy link
Contributor Author

aah. Closing this.

@sumeerbhola sumeerbhola changed the title TestIndexedBatchMutation failed on Windows Test failure on Windows Jan 3, 2023
@sumeerbhola sumeerbhola reopened this Jan 3, 2023
@sumeerbhola
Copy link
Collaborator

sumeerbhola commented Jan 3, 2023 •
edited
Loading

Reopened this issue, so we can add windows flakes here. Like the earlier report, I noticed

=== RUN   TestIndexedBatchMutation
    batch_test.go:401: 
        testdata/indexed_batch_mutation:2:

And looking at the raw logs

2023-01-03T14:03:56.8663893Z === RUN   TestBytesIterated/Uncompressed
2023-01-03T14:03:56.8663978Z fatal error: out of memory
2023-01-03T14:03:56.8663992Z 
2023-01-03T14:03:56.8664081Z goroutine 3220 [running]:
2023-01-03T14:03:56.8664184Z runtime.throw({0x7ff6c4b300e0?, 0x1?})
2023-01-03T14:03:56.8664459Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/runtime/panic.go:992 +0x76 fp=0xc0c06c1918 sp=0xc0c06c18e8 pc=0x7ff6c45da716
2023-01-03T14:03:56.8664603Z github.com/cockroachdb/pebble/internal/manual.New(0x60)
2023-01-03T14:03:56.8664850Z 	D:/a/pebble/pebble/internal/manual/manual.go:45 +0x68 fp=0xc0c06c1948 sp=0xc0c06c1918 pc=0x7ff6c48c0288
2023-01-03T14:03:56.8665056Z github.com/cockroachdb/pebble/internal/cache.(*entryAllocCache).alloc(0x7ff6c4a6d8a0?)
2023-01-03T14:03:56.8665305Z 	D:/a/pebble/pebble/internal/cache/entry_normal.go:82 +0x2f fp=0xc0c06c1968 sp=0xc0c06c1948 pc=0x7ff6c48c354f
2023-01-03T14:03:56.8665463Z github.com/cockroachdb/pebble/internal/cache.entryAllocNew()
2023-01-03T14:03:56.8665697Z 	D:/a/pebble/pebble/internal/cache/entry_normal.go:40 +0x3e fp=0xc0c06c19a0 sp=0xc0c06c1968 pc=0x7ff6c48c331e
2023-01-03T14:03:56.8665935Z github.com/cockroachdb/pebble/internal/cache.newEntry(0xc1a7a7dcd0, {{0x27cfd5f2f10?, 0xc000100d00?}, 0x200000003?}, 0x62)
2023-01-03T14:03:56.8666167Z 	D:/a/pebble/pebble/internal/cache/entry.go:67 +0x4a fp=0xc0c06c1a48 sp=0xc0c06c19a0 pc=0x7ff6c48c2fca
2023-01-03T14:03:56.8666401Z github.com/cockroachdb/pebble/internal/cache.(*shard).Set(0xc1a7a7dcd0, 0x67?, 0x67?, 0x27cfd5f2ef0?, 0x27cfd5f2ef0)
2023-01-03T14:03:56.8666702Z 	D:/a/pebble/pebble/internal/cache/clockpro.go:148 +0x265 fp=0xc0c06c1ae0 sp=0xc0c06c1a48 pc=0x7ff6c48c08e5
2023-01-03T14:03:56.8666935Z github.com/cockroachdb/pebble/internal/cache.(*Cache).Set(0xc1a7a7dce0?, 0x27cfd5f2f10?, 0x27cfd5f2f10?, 0x0?, 0x21?)
2023-01-03T14:03:56.8667174Z 	D:/a/pebble/pebble/internal/cache/clockpro.go:750 +0x46 fp=0xc0c06c1b38 sp=0xc0c06c1ae0 pc=0x7ff6c48c2766
2023-01-03T14:03:56.8667389Z github.com/cockroachdb/pebble/sstable.(*Reader).readBlock(0xc1a8208c00, {0x5?, 0x5?}, 0x0, 0xc0001816d8, 0x0)
2023-01-03T14:03:56.8667618Z 	D:/a/pebble/pebble/sstable/reader.go:3074 +0x58c fp=0xc0c06c1c30 sp=0xc0c06c1b38 pc=0x7ff6c4938dac
2023-01-03T14:03:56.8667815Z github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).readBlockWithStats(...)
2023-01-03T14:03:56.8667947Z 	D:/a/pebble/pebble/sstable/reader.go:629
2023-01-03T14:03:56.8668151Z github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).loadBlock(0xc000181400, 0x5?)
2023-01-03T14:03:56.8668377Z 	D:/a/pebble/pebble/sstable/reader.go:532 +0x28a fp=0xc0c06c1cd0 sp=0xc0c06c1c30 pc=0x7ff6c4930e2a
2023-01-03T14:03:56.8668581Z github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).firstInternal(0xc000181400)
2023-01-03T14:03:56.8668807Z 	D:/a/pebble/pebble/sstable/reader.go:1074 +0xee fp=0xc0c06c1d40 sp=0xc0c06c1cd0 pc=0x7ff6c493284e
2023-01-03T14:03:56.8668993Z github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).First(0xc000181400?)
2023-01-03T14:03:56.8669214Z 	D:/a/pebble/pebble/sstable/reader.go:1056 +0x2e fp=0xc0c06c1d60 sp=0xc0c06c1d40 pc=0x7ff6c493272e
2023-01-03T14:03:56.8669476Z github.com/cockroachdb/pebble/sstable.(*twoLevelCompactionIterator).skipForward(0xc19397dad0, 0x0?, {{0x0?, 0x0?, 0x7ff6c4aa66c0?}, 0x0?})
2023-01-03T14:03:56.8669695Z 	D:/a/pebble/pebble/sstable/reader.go:2458 +0xaa fp=0xc0c06c1dd0 sp=0xc0c06c1d60 pc=0x7ff6c493724a
2023-01-03T14:03:56.8669905Z github.com/cockroachdb/pebble/sstable.(*twoLevelCompactionIterator).Next(0xc19397dad0)
2023-01-03T14:03:56.8670129Z 	D:/a/pebble/pebble/sstable/reader.go:2417 +0x5e fp=0xc0c06c1e10 sp=0xc0c06c1dd0 pc=0x7ff6c493707e
2023-01-03T14:03:56.8670504Z github.com/cockroachdb/pebble/sstable.testBytesIteratedWithCompression(0xc180a7c340, 0x7ff6c4c4df8d?, 0x0, {0xc180a7a210, 0x5, 0x7ff6c4d9e320?}, {0xc0c06c1f38, 0x5, 0x7ff6c460d6f3?})
2023-01-03T14:03:56.8670740Z 	D:/a/pebble/pebble/sstable/reader_test.go:602 +0x32e fp=0xc0c06c1ef0 sp=0xc0c06c1e10 pc=0x7ff6c496fd0e
2023-01-03T14:03:56.8670935Z github.com/cockroachdb/pebble/sstable.TestBytesIterated.func2(0xc18082d1e0?)
2023-01-03T14:03:56.8671162Z 	D:/a/pebble/pebble/sstable/reader_test.go:629 +0x79 fp=0xc0c06c1f70 sp=0xc0c06c1ef0 pc=0x7ff6c4970139
2023-01-03T14:03:56.8671281Z testing.tRunner(0xc180a7c340, 0xc180a33e00)
2023-01-03T14:03:56.8671548Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1439 +0x102 fp=0xc0c06c1fc0 sp=0xc0c06c1f70 pc=0x7ff6c46b6cc2
2023-01-03T14:03:56.8671630Z testing.(*T).Run.func1()
2023-01-03T14:03:56.8671897Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1486 +0x2a fp=0xc0c06c1fe0 sp=0xc0c06c1fc0 pc=0x7ff6c46b7b6a
2023-01-03T14:03:56.8671981Z runtime.goexit()
2023-01-03T14:03:56.8672232Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0c06c1fe8 sp=0xc0c06c1fe0 pc=0x7ff6c460ae81
2023-01-03T14:03:56.8672321Z created by testing.(*T).Run
2023-01-03T14:03:56.8672513Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1486 +0x35f
2023-01-03T14:03:56.8672522Z 
2023-01-03T14:03:56.8672611Z goroutine 1 [chan receive]:
2023-01-03T14:03:56.8672778Z testing.(*T).Run(0xc00015cd00, {0x7ff6c4b32084?, 0x7ff6c460d6f3?}, 0x7ff6c4b52120)
2023-01-03T14:03:56.8672957Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1487 +0x37a
2023-01-03T14:03:56.8673068Z testing.runTests.func1(0xc00016dad0?)
2023-01-03T14:03:56.8673257Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1839 +0x6e
2023-01-03T14:03:56.8673424Z testing.tRunner(0xc00015cd00, 0xc000107cd8)
2023-01-03T14:03:56.8673612Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1439 +0x102
2023-01-03T14:03:56.8673808Z testing.runTests(0xc0001820a0?, {0x7ff6c4a74500, 0x48, 0x48}, {0x27ce4a10108?, 0x40?, 0x7ff6c4e347a0?})
2023-01-03T14:03:56.8673999Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1837 +0x457
2023-01-03T14:03:56.8674093Z testing.(*M).Run(0xc0001820a0)
2023-01-03T14:03:56.8674266Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1719 +0x5d9
2023-01-03T14:03:56.8674339Z main.main()
2023-01-03T14:03:56.8674442Z 	_testmain.go:221 +0x1aa
2023-01-03T14:03:56.8674450Z 
2023-01-03T14:03:56.8674542Z goroutine 3136 [chan receive]:
2023-01-03T14:03:56.8674702Z testing.(*T).Run(0xc180a7c000, {0x7ff6c4b2f49b?, 0xc129dd7524?}, 0xc180a33e00)
2023-01-03T14:03:56.8674887Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1487 +0x37a
2023-01-03T14:03:56.8675073Z github.com/cockroachdb/pebble/sstable.TestBytesIterated(0xc18082dd40?)
2023-01-03T14:03:56.8675231Z 	D:/a/pebble/pebble/sstable/reader_test.go:628 +0x11f
2023-01-03T14:03:56.8675348Z testing.tRunner(0xc180a7c000, 0x7ff6c4b52120)
2023-01-03T14:03:56.8675529Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1439 +0x102
2023-01-03T14:03:56.8675620Z created by testing.(*T).Run
2023-01-03T14:03:56.8675794Z 	C:/hostedtoolcache/windows/go/1.18.9/x64/src/testing/testing.go:1486 +0x35f
2023-01-03T14:03:56.8675801Z 
2023-01-03T14:03:56.8675892Z goroutine 59 [chan receive]:
2023-01-03T14:03:56.8676068Z github.com/cockroachdb/pebble/sstable.(*writeQueue).runWorker(0xc00025cb80)
2023-01-03T14:03:56.8676218Z 	D:/a/pebble/pebble/sstable/write_queue.go:98 +0x4c
2023-01-03T14:03:56.8676385Z created by github.com/cockroachdb/pebble/sstable.newWriteQueue
2023-01-03T14:03:56.8676529Z 	D:/a/pebble/pebble/sstable/write_queue.go:57 +0xd6
2023-01-03T14:03:56.8676665Z FAIL	github.com/cockroachdb/pebble/sstable	7.908s

@bananabrick
Copy link
Contributor Author

Another: https://pipelines.actions.githubusercontent.com/serviceHosts/f202191f-2284-4a45-9c08-b99fe51efaad/_apis/pipelines/1/runs/3808/signedlogcontent/18?urlExpires=2023-01-03T19%3A58%3A30.2133565Z&urlSigningMethod=HMACV1&urlSignature=xO8XevytftGcgG0YFMFbjJfDkKeAeAdG9uOrkjO3QA8%3D

@jbowens
Copy link
Collaborator

jbowens commented Jan 3, 2023

might have something to do with the increased memory usage seen here?

@nicktrav
Copy link
Contributor

nicktrav commented Jan 4, 2023

Some more datapoints:

All of these seem to have failed with something along the lines of:

2023-01-03T14:35:06.6273384Z runtime: VirtualAlloc of 2147483648 bytes failed with errno=1455
2023-01-03T14:35:06.6273483Z fatal error: out of memory

@nicktrav nicktrav changed the title Test failure on Windows windows: test flakes / failures due to out of memory Jan 6, 2023
nicktrav added a commit to nicktrav/pebble that referenced this issue Jan 6, 2023
@nicktrav
sstable: skip two known test flakes on windows
acae32f 
Skip two tests that are known to be flakes when run on windows.

Touches cockroachdb#1897.
@nicktrav nicktrav mentioned this issue Jan 6, 2023
Merged
nicktrav added a commit that referenced this issue Jan 6, 2023
@nicktrav
sstable: skip two known test flakes on windows
65ff304 
Skip two tests that are known to be flakes when run on windows.

Touches #1897.
@nicktrav nicktrav self-assigned this Jan 9, 2023
nicktrav added a commit to nicktrav/pebble that referenced this issue Jan 12, 2023
@nicktrav
Revert "sstable: skip two known test flakes on windows"
5c339d5 
This reverts commit 65ff304.

Touches cockroachdb#1897.
@nicktrav nicktrav mentioned this issue Jan 12, 2023
Merged
nicktrav added a commit that referenced this issue Jan 12, 2023
@nicktrav
Revert "sstable: skip two known test flakes on windows"
dac60ce 
This reverts commit 65ff304.

Touches #1897.
@nicktrav
Copy link
Contributor

Looking at recent master builds, I don't see the same rate of Windows flakes. Hopefully this is related to #2236.

Going to close this for now, but feel free to re-open if Windows builds start flaking out again.

@jbowens jbowens added this to Storage Jun 4, 2024
@jbowens jbowens moved this to Done in Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nicktrav nicktrav

Labels
A-storage T-storage
Projects
Storage
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nicktrav @jbowens @bananabrick @sumeerbhola