Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add persistence option to memcached on helm chart #13619

Merged
merged 5 commits into from
Sep 2, 2024

Conversation

AndreZiviani
Copy link
Contributor

@AndreZiviani AndreZiviani commented Jul 22, 2024

What this PR does / why we need it:
Add an option to configure persistence on memcached, very useful with extstore option.
More info on Grafana Blog post, based mostly on loki-distributed helm chart

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@AndreZiviani AndreZiviani requested a review from a team as a code owner July 22, 2024 19:07
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jul 22, 2024
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jul 22, 2024
@AndreZiviani
Copy link
Contributor Author

not sure why helm-ci / Test Helm Chart (pull_request) failed, seems unrelated to these changes

@AndreZiviani AndreZiviani force-pushed the memcached-persistence branch 2 times, most recently from ba6375d to c089234 Compare July 29, 2024 19:59
@AndreZiviani
Copy link
Contributor Author

@JStickler sorry to tag you directly but can I get a review here please?

@JStickler
Copy link
Contributor

@AndreZiviani I'm the technical writer on the Loki team, not a developer. So I do not review code, just docs PRs.

@AndreZiviani
Copy link
Contributor Author

my bad, can you tag the correct person to review this please?

@AndreZiviani AndreZiviani force-pushed the memcached-persistence branch 2 times, most recently from 86c208b to 3e192c2 Compare August 7, 2024 13:29
@AndreZiviani
Copy link
Contributor Author

@vlad-diachenko @DylanGuedes sorry to tag you directly but could you take a look here please? thanks

@AndreZiviani AndreZiviani force-pushed the memcached-persistence branch 2 times, most recently from caac9c3 to fd12ba3 Compare August 9, 2024 19:03
Comment on lines 125 to 133
{{- if .extraVolumeMounts }}
volumeMounts:
{{- if .persistence.enabled }}
- name: data
mountPath: {{ .persistence.mountPath }}
{{- end }}
{{- if .extraVolumeMounts }}
{{- toYaml .extraVolumeMounts | nindent 12 }}
{{- end }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this behave when there are neither .extraVolumeMounts and .persistence.enabled is false? I would assume it renders the key volumeMounts: but without list items. I assume this would cause invalid YAML.
In that case, it should either not render volumeMounts at all, or volumeMounts: [].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would assume it renders the key volumeMounts: but without list items

that is correct

I assume this would cause invalid YAML.

that is not an issue, for exemple env is empty by default but I've added another check just in case

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, interesting, I would expect that to be invalid. Nevertheless, thanks for the additional check.

Copy link
Contributor

@chaudum chaudum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other question I still have is:

Do we want to set the --extended CLI arguments on the memcached container that are required for using the extstore feature, if persistence is enabled?

I see that they could be set using the extraExtendedOptions, but maybe it would be nice if you don't have to remember the the options.

Wdyt?

@AndreZiviani
Copy link
Contributor Author

AndreZiviani commented Aug 30, 2024

Do we want to set the --extended CLI arguments on the memcached container that are required for using the extstore feature, if persistence is enabled?

I see that they could be set using the extraExtendedOptions, but maybe it would be nice if you don't have to remember the the options.

yeah... I thought about that, loki-distributed does something like that but that was an issue for me because that paremeter makes memcached use the disk for persistency which means that my performance was bound to disk speed and it is not possible to remove/disable it so I had to declare the volume outside of the helm chart. I used the persistency as a second tier (like swap) which makes the hot data fast but could store less frequently accessed data on disk, more details on the blog post.
I guess I could that and the other -o ext_path=/data/file parameter, which actually enables the storage backend, as values in order to allow people to override them, let me see what I can do

@AndreZiviani AndreZiviani force-pushed the memcached-persistence branch from 2831a1e to 2959d56 Compare August 30, 2024 16:44
@AndreZiviani
Copy link
Contributor Author

I tried doing in a more flexible way but it became too complex, I believe how I did now is enough

I also rebased the branch to fix the conflicts

Copy link
Contributor

@chaudum chaudum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@chaudum chaudum merged commit ef1df0e into grafana:main Sep 2, 2024
63 checks passed
@AndreZiviani AndreZiviani mentioned this pull request Sep 19, 2024
7 tasks
mraboosk pushed a commit to mraboosk/loki that referenced this pull request Oct 7, 2024
…a#13619)

Add an option to configure persistence on memcached, very useful with extstore option.

More info on Grafana Blog post (https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/), based mostly on `loki-distributed` Helm chart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/helm size/L type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants