Loki can write but cannot read data from s3 storage

[Alexsandr-Random]

Stack - Grafana + Loki + promtail + s3 storage (seaweedfs) - all latest versions
Loki could write logs into s3 storage (could download logs via aws cli and general size of bucket increasing)
But when using grafana dashboard (or just loki cli) - loki can not extract historical data from s3 (only logs from cache)
It is NOT s3 permissions issue (user got admin privileges, e.g. read, list, write, tag...)
It is NOT connectivity issue (because we could write logs)
it is NOT grafana issue (loki cli also can not query historical data from s3)
Seems like bug...

Here is my loki configuration:

auth_enabled: false
chunk_store_config:
#  max_look_back_period: 336h
  max_look_back_period: 0s

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  # Read timeout for HTTP server
  http_server_read_timeout: 5m
  # Write timeout for HTTP server
  http_server_write_timeout: 5m
  # Write timeout for HTTP server
  http_server_idle_timeout: 5m

limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 360h
#    retention_period: 336h

#
schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb-shipper
    object_store: s3
    schema: v11
    index:
      prefix: loki_
      period: 24h
#
storage_config:
 boltdb_shipper:
   active_index_directory: /loki/index
   cache_location: /loki/index_cache
   shared_store: s3
 #  resync_interval: 45s
 aws:
   bucketnames: <secret>
   endpoint: seaweedfs-filer.service.consul:8333
   region: eu
   access_key_id: <secret>
   secret_access_key: <secret>
   insecure: true
 #  sse_encryption: false 
   s3forcepathstyle: true
#
compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 1m
  retention_enabled: true
  retention_delete_delay: 2h 
  retention_delete_worker_count: 150
#
ingester:
  chunk_idle_period: 3m
  chunk_block_size: 262144
  chunk_retain_period: 1m
  max_transfer_retries: 0
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  wal:
    dir: /loki/wal
#
table_manager:
    retention_deletes_enabled: true
    retention_period: 336h

[naveenpg11]

Can you share if you get any error message in read path (querier) ? @Alexsandr-Random

[Alexsandr-Random]

@naveenpg11 log level set to info (also turned on debugging, same result). When querying historical data - i don`t see any error\warn messages only info, like that:

level=info ts=2023-09-14T06:41:21.54833623Z caller=engine.go:234 component=querier org_id=fake traceID=2ac7087be700a0cc msg="executing query" type=range query="{application=\"API-MARKET\"}" length=30m0s step=5m0s query_hash=3362472753
level=info ts=2023-09-14T06:41:21.548577737Z caller=metrics.go:159 component=querier org_id=fake traceID=2ac7087be700a0cc latency=fast query="{application=\"API-MARKET\"}" query_hash=3362472753 query_type=limited range_type=range length=30m0s start_delta=110h41m21.548572637s end_delta=110h11m21.548572828s step=5m0s duration=193.015µs status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=25.939µs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s
level=info ts=2023-09-14T06:41:21.549170547Z caller=engine.go:234 component=querier org_id=fake traceID=2ac7087be700a0cc msg="executing query" type=range query="{application=\"API-MARKET\"}" length=30m0s step=5m0s query_hash=3362472753
....
(repeated many times)

[dlang-geosphereat]

We are having the same issues. This is probably a duplicate of #10529

[chaudum]

We are having the same issues. This is probably a duplicate of #10529

Right. Closing this issue, because it's a duplicate. Please refer to the other issue if you have more questions.