roachtest: test recovery from out of disk after removing ballast file

[a-robinson]

In #22235 (comment), I tested out how some new code behaved in a small cluster that was running out of disk. In order to make some nodes fuller than others, I put 5GB ballast files on their disks. After the cluster filled up, I expected to be able to recover it by shrinking the ballast file down to progressively smaller sizes, but was surprised that even removing 1GB from each node's file wasn't enough for the cluster to recover.

See the comment linked above for more detail, but my takeaway was that rocksdb compactions require more scratch disk space than I would have expected, filling up a node's disk that had hundreds of megabytes of free space despite there being no writes happening on the cluster beyond the normal background load of timeseries data and node liveness heartbeats.

We should better understand this behavior and see if there's a good way to make rocksdb more conscious of the available space on disk when compacting.

I have tarballs of two of the data directories that filled up while doing rocksdb compactions saved for inspecting if anyone wants to take a look. Beware, they're 7.6GB each:
https://storage.googleapis.com/cockroach-alex/alex-disk-0005.tar.gz
https://storage.googleapis.com/cockroach-alex/alex-disk-0006.tar.gz

[petermattis]

My expectation is that a RocksDB compaction may temporarily require as much disk space as the input sstables. It sounds like more was being required which is surprising to me as well.

[bdarnell]

Are we going to be able to do anything here for 2.0 or should we bump it to 2.1?

[petermattis]

I doubt we'll be able to fix anything, but that's unclear until we do some investigation. I think a small bit of investigation is warranted before 2.0 is released.

[neeral]

@a-robinson how big is the cockroach data directory partition, in comparison to the 5GB ballast file?

[a-robinson]

Each data partition was only 10GB.

[a-robinson]

From https://github.com/facebook/rocksdb/releases/tag/v5.13.1:

SstFileManager now can cancel compactions if they will result in max space errors. SstFileManager users can also use SetCompactionBufferSize to specify how much space must be leftover during a compaction for auxiliary file functions such as logging and flushing.

[tbg]

Folding into #7882.