cst/cache: tracker based trim #18786
Conversation
abhijat
force-pushed
the
enhancement/18759/tracker-based-trim
branch
from
261d8b8 to
2ae21b5
Compare
abhijat
force-pushed
the
enhancement/18759/tracker-based-trim
branch
12 times, most recently
from
403a63e to
66e146d
Compare
|
| }); | ||
| }); | ||
| }); | ||
| _tracker_sync_timer.arm(tracker_sync_period); |
There was a problem hiding this comment.
So the sync will be done every 6 hours in a non-blocking manner.
Maybe we should sync early if the tracker is empty (this could happen when we upgrade) or when we're low on space but tracker says that we should have space? WDYT?
There was a problem hiding this comment.
The sync is based on access_time_tracker::trim which is one-way right now, it will only ever remove items from the cache and never add new items, which originally made sense because it was expected to remove manually deleted files from the tracker.
Right now the only way to add entries to the tracker is to issue a get on the cache.
To populate an empty tracker, maybe the sync (and tracker's trim method) should have an optional flag to enable it to add items in addition to removing them, IE when the flag argument is off by default but when it is on, if the item is present in incoming set but not in tracker, then it is added. If the item is not in the incoming set but present in tracker, then it is removed.
There was a problem hiding this comment.
when we're low on space but tracker says that we should have space
Would this happen because of extra tmp files and the index and tx files which are present on disk but not in the tracker?
Even if we sync the tracker state with disk in an additive manner, we would probably skip adding these files to the tracker, so the size difference might still be there after the sync.
There was a problem hiding this comment.
To populate an empty tracker, maybe the sync (and tracker's trim method) should have an optional flag to enable it to add items in addition to removing them
maybe the tracker could have a field which is populated when the tracker is loaded, if the tracker was upgraded from v1 to v2 and lost all data then we should set the flag. And if the flag is set we could start the sync early.
There was a problem hiding this comment.
Would this happen because of extra tmp files and the index and tx files which are present on disk but not in the tracker?
because of this or because redpanda added some files to the cache and crashed before the tracker was saved to disk
There was a problem hiding this comment.
On startup in cleanup_at_start we sync the tracker. I added a flag there to do add entries to tracker if they are found in directory but missing in the data structure (except index, tx and tmp files). I think this should take care of both scenarios:
- empty tracker due to upgrade - the sync is done after read from disk so we will populate the tracker during sync with on-disk data
- tracker not written during shutdown - the sync will write missing entries into the tracker on startup
abhijat
force-pushed
the
enhancement/18759/tracker-based-trim
branch
8 times, most recently
from
0ffdab3 to
bea55ac
Compare
abhijat
force-pushed
the
enhancement/18759/tracker-based-trim
branch
2 times, most recently
from
92fc40c to
a0c3319
Compare
* The access time tracker now also tracks file sizes. * The key to the tracker map is now the full file path instead of a hash * The serialized form of tracker now contains a version to distinguish between the old and new data structures. * The recursive directory walker uses tracker entries to avoid file stat operations.
When trimming a first attempt is made to get LRU entries from the access tracker. If this does not free up enough space, then we proceed with the existing trim based on directory walk.
The tracker is synced periodically with on-disk data using a directory walk. This is necessary because once we switch to trimming using the tracker, the data in tracker and on-disk may drift because of tmp files, aborted downloads or manual deletion of files. A periodic sync brings the tracker in sync with the metadata on disk.
abhijat
force-pushed
the
enhancement/18759/tracker-based-trim
branch
from
a0c3319 to
c17d79d
Compare
|
new failures in https://buildkite.com/redpanda/redpanda/builds/50148#01900b33-65e0-45e5-ac78-fda6b6125a41: |
|
/ci-repeat |
|
/backport v24.1.x |
This changeset extends the access time tracker to hold size as well as atime of cache entries. The keys for the tracker are changed from hash of paths to paths themselves.
The entries in the cache are used directly to trim, by sorting the paths based on atime and then removing the least recently used entries. This avoids the need to make directory listing or file stat calls during the trim.
A background task is added to periodically sync the tracker with on-disk data.
FIXES: #18759
New metrics:
in_mem_trims- number of tracker based trimstracker_syncs- number of times tracker was synced with disktracker_size- size of tracker (count of entries - one per file - in cache)Backports Required
Release Notes