test_download_remote_layers_api flakyness

[koivunej]

https://neon-github-public-dev.s3.amazonaws.com/reports/pr-3818/debug/4436920822/index.html#suites/b97efae3a617afb71cb8142f5afa5224/ea27292fb12d954/:

AssertionError: current_physical_size is sum of loaded layer sizes, independent of whether local or remote
assert 29835264 == 29417472
  +29835264
  -29417472

From the test it looks this could be caused by wal being received from safekeepers after measuring it. If not, then it's probably something mystical like #3209.

[arssher]

duplicate of #3422

[jcsp]

A recent failure of this test: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-5207/6157084684/index.html#suites/b97efae3a617afb71cb8142f5afa5224/9b22f1b52bf4c315

AssertionError: we redownloaded all the layers
assert 29835264.0 == 29876224.0

[koivunej]

Hehe, it's an issue I have created. Today's analysis:

TL;DR: it definitelty triggers the path I added in #5233.

An instance today which also has a succesful run.

The test log does have the fixup I added:

2023-09-15 04:56:16.274 INFO [test_ondemand_download.py:402] fixing up filled_current_physical from 29835264 to 29876224 (40960)

it seems in this case it works wrong:

AssertionError: we redownloaded all the layers
assert 29835264.0 == 29876224.0

I don't understand. Thought I think without my change in #5233, it would had failed on the previous assert near the "fixing up", so it might be my "fixing up" is only partial.

EDIT: Log analysis later on: logs clearly show inmemory flush after metrics were read during shutdown.

[koivunej]

Finally closing this due to my above log analysis and being able to merge #5322 as it fixes that exact problem.