"ignoring DynamoDB URL path" #310

setevoy2 · 2019-02-11T13:49:07Z

Hi.
I'm trying to configure an S3 bucket and DynamoDB's table to be used from Loki using two examples - here and here.

The first one says:

schema_config:
  configs:
  - from: 0
    store: aws
    object_store: aws
    schema: v9
    index:
      prefix: dynamodb_table_name
      period: 0

And the second one is:

  configs:
  - from: 0
    store: dynamo
    object_store: s3
    schema: v9
    index:
      prefix: dynamodb_table_name
      period: 0

But using both examples - I'm getting the error:

loki_1 | level=warn ts=2019-02-11T13:25:05.401742684Z caller=factory.go:133 msg="ignoring DynamoDB URL path" path=W8/+CWFN@eu-west-1
loki_1 | level=warn ts=2019-02-11T13:25:05.401758262Z caller=dynamodb_storage_client.go:833 msg="ignoring DynamoDB URL path" path=W8/+CWFN@eu-west-1

Then once any request made from Grafana - Loki going down with another error:

loki_1 | panic: runtime error: invalid memory address or nil pointer dereference
loki_1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe5809e]
loki_1 |
loki_1 | goroutine 171 [running]:
loki_1 | github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws.dynamoDBRequestAdapter.Retryable(0xc0000a8000, 0x12822fa)
loki_1 | /go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws/dynamodb_storage_client.go:457 +0xe
loki_1 | github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws.dynamoDBStorageClient.queryPage(0xc00009ae00, 0x4000000000000000, 0x0, 0x0, 0x0, 0x186a0, 0x3ff4cccccccccccd, 0x12b1ab6, 0x51, 0x12b3d66, ...)
loki_1 | /go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws/dynamodb_storage_client.go:380 +0x3b3
loki_1 | github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws.dynamoDBStorageClient.query(0xc00009ae00, 0x4000000000000000, 0x0, 0x0, 0x0, 0x186a0, 0x3ff4cccccccccccd, 0x12b1ab6, 0x51, 0x12b3d66, ...)
loki_1 | /go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws/dynamodb_storage_client.go:337 +0x44a
loki_1 | github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws.dynamoDBStorageClient.query-fm(0x1401d00, 0xc0007576b0, 0xc0003f4a80, 0x8, 0xc000444b80, 0x14, 0x0, 0x0, 0x0, 0x0, ...)
loki_1 | /go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/aws/dynamodb_storage_client.go:285 +0xb4
loki_1 | github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/util.DoParallelQueries.func1(0x1401d00, 0xc000757500, 0xc00075a240, 0xc00075a2a0, 0xc000132400, 0xc0002abba0)
loki_1 | /go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/util/util.go:42 +0x111
loki_1 | created by github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/util.DoParallelQueries
loki_1 | /go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/util/util.go:34 +0xf4
admin_loki_1 exited with code 2

Can't understand this message:

msg="ignoring DynamoDB URL path" path=W8/+C***WFN@eu-west-1

Why it's starting from the "W8/+" if inside config it looks like:

...
schema_config:
  configs:
  - from: 0
    store: dynamo
    object_store: s3
    schema: v9
    index:
      prefix: loki-poc
      period: 0

storage_config:
#  boltdb:
#    directory: /tmp/loki/index

#  filesystem:
#    directory: /tmp/loki/chunks
  aws:
     s3: s3://AKI***PHA:0/W8/+***WFN@eu-west-1/logger-loki-poc
     dynamodbconfig:
       dynamodb: "dynamodb://AKI***PHA:0/W8/+***WFN@eu-west-1"

The SECRET_KEY is stating from the "0/W8/+" symbols - not "W8/+C" - where the "0/" symbol missed? Mybe a bug with "0/"? Or just my wrong configuration?

DymanoDB table "loki-poc" was created manually, with the OrgID as a primary key filed (not sure this is correct).

Trying to add the quotes around it - but didn't help.

The text was updated successfully, but these errors were encountered:

setevoy2 · 2019-02-11T14:20:20Z

Yes, the issue above caused by the "0/" and "/" symbols in the SECRET_KEY.
Generated new key without "/" - now it "works" but with another error which is obviously caused by my DynamoDB table's config:

loki_1 | level=error ts=2019-02-11T14:08:35.322757196Z caller=chunk_store.go:391 org_id=fake msg="error querying storage" err="QueryPage error: table=loki-poc, err=ValidationException: Query condition missed key schema element: OrgID\n\tstatus code: 400, request id: 7INPECMU"

yubozhao · 2019-02-12T18:55:18Z

Hi @setevoy2 Please refer documents from the master branch with the latest commit. So use operations.md as the reference. The secret_key bug you can file it at github.com/weaveswork/common. That's the upstream package that handles this.

setevoy2 · 2019-02-13T10:44:40Z

Thanks, @yubozhao!

Indeed - there are keys mentioned (now?) in the operations.md:

primary index key to h (string) and use r (binary) as the sort key
Now it works.

Not sure about the 0/ bug (from the http://github.com/weaveworks/common) but anyway it works with new keys, so it's OK for now.

base698 · 2019-04-28T17:06:31Z

I'm getting level=error ts=2019-04-28T17:01:44.281886806Z caller=client.go:186 msg="final error sending batch" status=401 error="server returned HTTP status 401 Unauthorized (401): no org id" in promtail. Is this from Loki or Dynamodb?

yubozhao · 2019-04-28T18:37:13Z

@base698
I haven't touch Loki for a while, so my information might be outdated. So my last understandinf of Loki is Loki does have basic multi-tenant support, but I don't think it is documented yet. If you look closely in their production deployment example, it has a nginx sits in front of Loki and attaches a new X-Scope-OrgID header in the incoming requests.

You can do the same thing as Loki's team with another layer in front of Loki and mutate the incoming requests with the new header.

yubozhao · 2019-04-28T18:38:45Z

You can also do auth_enabled: false in the Loki's configuration setting.

This is where they talked about it https://github.com/grafana/loki/blob/d33722dc71559a1cfd150eb4926b073537ce4c5c/docs/operations.md#authentication

base698 · 2019-04-28T18:46:10Z

`auth_enabled: false` results in a 400 error "sample missing metric name" which is in the code here: https://github.com/grafana/loki/blob/master/vendor/github.com/cortexproject/cortex/pkg/util/validation/validate.go#L16 It works locally with a config that doesn't have the AWS stuff in it. 🤷

…

On Sun, Apr 28, 2019 at 11:38 AM Bozhao ***@***.***> wrote: You can also do auth_enabled: false in the Loki's configuration setting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#310 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABSVGXEQXAZ3G5LHKATB5DPSXVLZANCNFSM4GWQ4SLQ> .

yubozhao · 2019-04-28T20:41:49Z

Hmm interesting. I have ours running on k8s/aws with dynamodb/s3. I don’t think I did anything different. Maybe an older version? Keep me posted with your progress. Want to learn

base698 · 2019-04-28T20:59:08Z

Yeah I've been digging through the code. It's looking for metric name in the label to have __name__ which is not sent by promtail.

…

On Sun, Apr 28, 2019, 1:41 PM Bozhao ***@***.***> wrote: Hmm interesting. I have ours running on k8s/aws with dynamodb/s3. I don’t think I did anything different. Maybe an older version? Keep me posted with your progress. Want to learn — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#310 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABSVGQL24AGUMUI4FNYV7LPSYDZLANCNFSM4GWQ4SLQ> .

base698 · 2019-04-28T21:07:20Z

In this file: https://github.com/grafana/loki/blob/master/pkg/promtail/targets/filetargetmanager.go#L171

Adding:

			labels = labels.Merge(model.LabelSet{"__name__": model.LabelValue("somename")})

Fixes it. I see the logs now in loki

yubozhao · 2019-04-28T21:32:59Z

Cool glad you figured it out. Enjoy your Sunday!

stale · 2019-09-03T16:10:36Z

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

Sreevani871 · 2020-02-13T11:28:44Z

auth_enabled: false results in a 400 error "sample missing metric name" which is in the code here: https://github.com/grafana/loki/blob/master/vendor/github.com/cortexproject/cortex/pkg/util/validation/validate.go#L16 It works locally with a config that doesn't have the AWS stuff in it. 🤷
…
On Sun, Apr 28, 2019 at 11:38 AM Bozhao @.***> wrote: You can also do auth_enabled: false in the Loki's configuration setting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#310 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AABSVGXEQXAZ3G5LHKATB5DPSXVLZANCNFSM4GWQ4SLQ .

In this file: https://github.com/grafana/loki/blob/master/pkg/promtail/targets/filetargetmanager.go#L171

Adding:
			labels = labels.Merge(model.LabelSet{"__name__": model.LabelValue("somename")})
Fixes it. I see the logs now in loki

Facing the same issue. I didn't understand where to add this line of code labels = labels.Merge(model.LabelSet{"__name__": model.LabelValue("somename")}) exactly. https://github.com/grafana/loki/blob/master/pkg/promtail/targets/filetargetmanager.go#L171 does not have any labels variable.

Update from upstream repository

garo mentioned this issue Apr 5, 2019

Support using standard AWS credential discovery instead of hardcoded tokens #451

Closed

yubozhao mentioned this issue Apr 28, 2019

Allow promtail to set X-Scope-OrgID header dynamically #259

Closed

stale bot added the stale A stale issue or PR that will automatically be closed. label Sep 3, 2019

stale bot closed this as completed Sep 10, 2019

periklis added a commit to periklis/loki that referenced this issue Sep 10, 2024

Merge pull request grafana#310 from grafana/main

8d84e6a

Update from upstream repository

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"ignoring DynamoDB URL path" #310

"ignoring DynamoDB URL path" #310

setevoy2 commented Feb 11, 2019

setevoy2 commented Feb 11, 2019

yubozhao commented Feb 12, 2019

setevoy2 commented Feb 13, 2019

base698 commented Apr 28, 2019

yubozhao commented Apr 28, 2019

yubozhao commented Apr 28, 2019 •

edited

Loading

base698 commented Apr 28, 2019 via email •

edited

Loading

yubozhao commented Apr 28, 2019

base698 commented Apr 28, 2019 via email

base698 commented Apr 28, 2019

yubozhao commented Apr 28, 2019

stale bot commented Sep 3, 2019

Sreevani871 commented Feb 13, 2020

"ignoring DynamoDB URL path" #310

"ignoring DynamoDB URL path" #310

Comments

setevoy2 commented Feb 11, 2019

setevoy2 commented Feb 11, 2019

yubozhao commented Feb 12, 2019

setevoy2 commented Feb 13, 2019

base698 commented Apr 28, 2019

yubozhao commented Apr 28, 2019

yubozhao commented Apr 28, 2019 • edited Loading

base698 commented Apr 28, 2019 via email • edited Loading

yubozhao commented Apr 28, 2019

base698 commented Apr 28, 2019 via email

base698 commented Apr 28, 2019

yubozhao commented Apr 28, 2019

stale bot commented Sep 3, 2019

Sreevani871 commented Feb 13, 2020

yubozhao commented Apr 28, 2019 •

edited

Loading

base698 commented Apr 28, 2019 via email •

edited

Loading