Dynamic metadata filter not working as described

[valodzka]

Nomad version

1.5.2 / linux

Issue

I've tried example described here https://developer.hashicorp.com/nomad/api-docs/task-api and found that filter not working with dynamic metadata.

Reproduction steps

nomad node meta apply example=a
nomad node status -filter 'Meta.example == "a"'
Error querying node status: Unexpected response code: 400 (failed to read result page: error finding value in datum: /Meta/example at part 1: couldn't find key "example")

[schmichael]

The error is poor here, but I suspect what you're running into is that there's a delay between setting node metadata and when it is available via nomad node status.

The API docs mentions this, but I forgot to add CLI docs! Fixing that in #16828

[valodzka]

Hi @schmichael

I don't think update delay is the reason. 5 days ago after meta update:

nomad node status -filter 'Meta.example == "a"'
Error querying node status: Unexpected response code: 400 (failed to read result page: error finding value in datum: /Meta/example at part 1: couldn't find key "example")

For context: i'm running this in simple cluster with 3 nodes (all nodes working as server and as client). It works as expected with nomad agent -dev, but not in cluster.

[schmichael]

You're right @valodzka! I was able to reproduce the error with these agent configs and running your commands.

I can see the example = a metadata when using nomad node status -verbose on the node in question, so it is getting set properly... something seems to be going wrong with the filter processing. Hitting the HTTP API directly produces the same results:

> GET http://127.0.0.1:4646/v1/nodes?filter=Meta.example+%3D%3D+%22a%22
* Sending request and receiving response...
< HTTP/1.1 400 Bad Request
< Vary: Accept-Encoding
< Date: Tue, 11 Apr 2023 00:01:07 GMT
failed to read result page: error finding value in datum: /Meta/example at part 1: couldn't find key "example"```

[schmichael]

This is a limitation of the underlying github.com/hashicorp/go-bexpr library we (and Consul) use for filters. While we try to find a fix that doesn't involve backwards incompatibilities there is a workaround:

"example" in Meta and Meta["example"] == "foo"

Performing an in test first will cause Nodes without the Meta["example"] key to be skipped instead of trying to perform the equality and failing to find the field.

[schmichael]

Fixed and will be going out in our next releases! Thanks for the report @valodzka !

[lattwood]

Looks like this was only fixed for the case of top level entries in Meta.

Example:

Meta
connect.log_level         = info
connect.proxy_concurrency = 1

> nomad node status -json -filter 'Meta.connect.proxy_concurrency == 1'
Error querying node status: Unexpected response code: 400 (failed to read result page: error finding value in datum: /Meta/connect/proxy_concurrency at part 1: couldn't find key "connect")
> nomad node status -json -filter '"connect.proxy_concurrency" in Meta and Meta["connect"]["proxy_concurrency"] == 1'
Error querying node status: Unexpected response code: 400 (failed to read result page: error finding value in datum: /Meta/connect/proxy_concurrency at part 1: couldn't find key "connect")

As you can see, the workaround didn't work.