Bug: Using Helm chart ruler ignores s3 backend

[swbsf]

What is the bug?

Hi,
Using mimir-distributed chart version 5.6.0. It seems ruler ignores the s3 block configuration for its storage, or it can't find the rule files for some reason. While Mimir tsdb use its bucket to store metrics. I'm quite confident this is a silly problem, but already spent quite some time on it. Maybe someone can help?

How to reproduce it?

Here is the simple configuration I am using:

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          region: sbg
          endpoint: s3.sbg.io.cloud.ovh.net
          access_key_id: {{ env "S3_ACCESS_KEY" }}
          secret_access_key: {{ env "S3_SECRET_KEY" }}
    blocks_storage:
      s3:
        bucket_name: f366bb575a03bca2c-mimir-tsdb
    alertmanager_storage:
      s3:
        bucket_name: f366bb575a03bca2c-mimir-alertmanager
    ruler_storage:
      s3:
        bucket_name: f366bb575a03bca2c-mimir-ruler

Added a rule files (generated from kubernetes-mixins) in the bucket. And in bucket/1/rules.yaml. "1" being the name of my first tenant.
Maybe my files are in the wrong place? I can't figure where they should be then.

What did you think would happen?

ruler should load rule files.

What was your environment?

mimir-distributed chart 5.6.0, Kubernetes 1.31

Any additional context to share?

Ruler logs

ts=2025-01-22T16:41:38.827332396Z caller=main.go:225 level=info msg="Starting application" version="(version=2.15.0, branch=HEAD, revision=24e4281)"
ts=2025-01-22T16:41:38.82939314Z caller=server.go:360 level=info msg="server listening on addresses" http=:8080 grpc=:9095
ts=2025-01-22T16:41:38.848767Z caller=memberlist_client.go:446 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-ruler-798bb676db-mgfg7-06b1ba4b
ts=2025-01-22T16:41:38.857222317Z caller=memberlist_client.go:563 level=info msg="memberlist fast-join starting" nodes_found=12 to_join=8
ts=2025-01-22T16:41:38.865743487Z caller=mapper.go:52 level=info msg="cleaning up mapped rules directory" path=/data
ts=2025-01-22T16:41:38.869118573Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2025-01-22T16:41:38.869066048Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2025-01-22T16:41:38.869845444Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2025-01-22T16:41:38.870080424Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2025-01-22T16:41:38.870095902Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2025-01-22T16:41:38.888277931Z caller=memberlist_client.go:583 level=info msg="memberlist fast-join finished" joined_nodes=8 elapsed_time=35.414325ms
ts=2025-01-22T16:41:38.888343063Z caller=memberlist_client.go:595 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.monitoring.svc.cluster.local.:7946
ts=2025-01-22T16:41:38.935505186Z caller=memberlist_client.go:602 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=12 elapsed_time=47.132694ms
ts=2025-01-22T16:41:39.634107987Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2025-01-22T16:41:39.634190973Z caller=module_service.go:82 level=info msg=starting module=server
ts=2025-01-22T16:41:39.634248224Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2025-01-22T16:41:39.634444467Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2025-01-22T16:41:39.635156065Z caller=module_service.go:82 level=info msg=starting module=store-queryable
ts=2025-01-22T16:41:39.6351962Z caller=module_service.go:82 level=info msg=starting module=ingester-ring
ts=2025-01-22T16:41:39.636262385Z caller=module_service.go:82 level=info msg=starting module=distributor-service
ts=2025-01-22T16:41:39.636395632Z caller=module_service.go:82 level=info msg=starting module=ruler
ts=2025-01-22T16:41:39.63668116Z caller=basic_lifecycler.go:299 level=info msg="instance not found in the ring" instance=mimir-ruler-798bb676db-mgfg7 ring=ruler
ts=2025-01-22T16:41:39.637039987Z caller=ruler.go:449 level=info msg="waiting until ruler is JOINING in the ring"
ts=2025-01-22T16:41:39.782069051Z caller=ruler.go:453 level=info msg="ruler is JOINING in the ring"
ts=2025-01-22T16:41:39.782129832Z caller=ruler.go:573 level=info msg="syncing rules" reason=initial
ts=2025-01-22T16:41:40.844166319Z caller=ruler.go:462 level=info msg="waiting until ruler is ACTIVE in the ring"
ts=2025-01-22T16:41:40.966112385Z caller=ruler.go:466 level=info msg="ruler is ACTIVE in the ring"
ts=2025-01-22T16:41:40.966161901Z caller=ruler.go:471 level=info msg="ruler is only now starting to evaluate rules"
ts=2025-01-22T16:41:40.966180713Z caller=ruler.go:525 level=info msg="ruler up and running"
ts=2025-01-22T16:41:40.966284852Z caller=mimir.go:897 level=info msg="Application started"