ethdb/pebble: sync pebble writes

[holiman]

This is likely the culprit behind several data corruption issues, e.g. where data has been written to the freezer, but the deletion from leveldb does not go through due to process crash.

Pebble has pretty strange sync options:

• sync=true: writes to the WAL are flushed to OS, and fsync:ed to disk.
• sync=false: writes to the WAL are in-mem, and flushed to OS at some future time.

Out of durability guarantees,

1. Even if the OS crashes, written batches are written
2. Even if process crashes, written batches are written
3. Batches are written sometime, at latest on controlled shutdown.

With leveldb, we could choose to land on 2: survive process crash, but potentially corrupt data on os crash. With pebble, it's either option 1 or 3. Current master uses 3.

This PR uses option 1.

Benchmarking, this PR is on 07, master on 08:

Jun 20 10:48:12 bench07.ethdevops.io geth INFO [06-20|08:48:12.787] Starting peer-to-peer node instance=Geth/v1.12.1-unstable-04040fcb-20230620/linux-amd64/go1.20.5
Jun 20 10:48:21 bench08.ethdevops.io geth INFO [06-20|08:48:21.775] Starting peer-to-peer node instance=Geth/v1.12.1-unstable-b1ef0bfe-20230619/linux-amd64/go1.20.5

[holiman]

Some assorted charts. This PR is yellow, master green

Somehow this PR managed to do faster ingress

No loss in disk write speed

All in all, this PR was about 1 hour faster syncing

So, I don't believe this PR makes things faster, but the important point is that at least there's no evidence of this change incurring any massive slowdown.

[rjl493456442]

More info in this ticket cockroachdb/pebble#2624