Feature: Add tenant_folders option to filesystem store

[jfolz]

What this PR does / why we need it:
Currently the docs for filesystem chunk storage state that chunks for each tenant will be stored in their own folder, but the Base64Encoder function that is used to encode chunk keys prevents this from happening.
Simply fixing the encoder function would break existing deployments, so instead this PR adds a Boolean option tenant_folders to the filesystem store that defaults to false.
Enabling this option means the new TenantBase64Encoder function is used, which is aware of the structure of chunk keys and preserves the tenant folder prefix.

Which issue(s) this PR fixes:
Fixes #4291

Special notes for your reviewer:
This is my first time working with the Loki source, but it seems like the more sensible solution would be to let the stores themselves handle how keys are encoded. Filesystem seems to be the only store to do any kind of encoding anyway. That would mean touching more of the code though, so I opted not do that. However, that meant I had to use the reflect package during client creation to avoid a circular import, which feels nasty.

Checklist

• Documentation added
• Tests updated

[CLAassistant]

All committers have signed the CLA.

[KMiller-Grafana]

Please make small wording changes suggested. After that, the documentation portion of this PR looks good to me.

[jfolz]

Any idea how to rerun the checks? docker-arm64 is marked as failed with all steps succeeding and previous runs with no code changes worked.

[KMiller-Grafana]

Thanks for making the changes. Docs look good to me now.

[owen-d]

Nice work! I definitely like associating the Encoder with the storage backend. I've left a few suggestions, but this is looking great, especially the considerations for upgrading.

[jfolz]

@owen-d Made the changes as discussed. Fully moving everything related to chunk encoding into the client would simplify some things and is probably the better solution long term, but this PR is getting quite big as it is ;)

[owen-d]

Looks great, but I realized an external problem we have :( . Because we can only specify one filesystem storage configuration, there's no way for users to actually upgrade; it'll only work on new Loki deployments or ones which haven't used the file system as a backend. Toggling this field will make all previous data unreadable as the expected paths no longer align :( .

However, I have wanted us to add support for multiple storage configurations of the same type for a while now. For instance, two s3 backends used in different schema configurations, or in our case, two filesystem ones. If we added support for that first, we could comfortably merge this as the upgrade path would be as simple as adding a new filesystem storage backend and a corresponding schema_config entry.

The two other ways I can see to do this are rather undesirable:

1. Make this part of the schema_config which can be rotated, but I dislike putting filesystem specific configs there.
2. Let the filesystem check both encoding styles on reads. I dislike this for performance reasons.

[jfolz]

Something I considered early on was to check during creation whether there are any files in the chunk directory that don't match the current settings and rename/move them. After some deliberation I decided that doing this implicitly during startup would be a bad idea, as it could take quite a while and doing things implicitly is bad. But it shouldn't be too difficult to provide this functionality as a small tool (similar to chunks-inspect) to manually upgrade and downgrade existing deployments. I see this as similar to changing the directory path. I wouldn't expect Loki to move the files for me, but it's possible for me to move them myself if I want to keep my existing data.

[owen-d]

I hear you, but I feel it's a trickier upgrade path than we want to support. Namely, toggling a field like this shouldn't prevent Loki from reading all past data, so I think we'll have to find a safer way to expose this or wait until we have the ability to specify multiple backends of the same type.

[owen-d]

Hey @jfolz, just wanted to let you know I haven't forgotten about this PR (and I really want it). I think we'll prioritize some work soon that we can piggyback this change onto.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[jfolz]

Dear Mr. Bot. While our filesystem hasn't exploded with too many files just yet, I would still very much appreciate if this or something like it is merged eventually.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[jfolz]

@owen-d can we mark this as keepalive?

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[jfolz]

.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[jfolz]

.

[jfolz]

Schema v12 (see #5291) solves the same issue and has a clear upgrade path. Thanks!