Clients querying not found endpoints aren't errors

[notnoop]

Nomad 0.11.0 marks 404 API queries log line with ERROR level. This is unexpected: A misbehaving API client can generate many ERROR log lines and potentially fill up server disks, causing denial-of-service. ERRORs should be reserved for issues that operators need to be alerted on and need to investigate.

A sample way to generate such log lines is the following queries on a new client:

$ curl http://localhost:4646/v1/allocation/b7d76831-4bcb-ec76-724b-ca0c6b45506b; echo
alloc not found
$ curl localhost:4646/v1/evaluation/6441fe30-d4e6-c5ef-b1f2-9fc917940d4c; echo
eval not found
$ curl localhost:4646/v1/client/allocation/614424f2-09f0-e64c-8f0c-cf824db519e4/stats; echo
Unknown allocation "614424f2-09f0-e64c-8f0c-cf824db519e4"

These generate the following log lines:

    2020-04-22T16:00:23.199-0400 [ERROR] http: request failed: method=GET path=/v1/allocation/b7d76831-4bcb-ec76-724b-ca0c6b45506b error="alloc not found" code=404
    2020-04-22T16:00:37.036-0400 [ERROR] http: request failed: method=GET path=/v1/evaluation/6441fe30-d4e6-c5ef-b1f2-9fc917940d4c error="eval not found" code=404
    2020-04-22T16:00:50.353-0400 [ERROR] http: request failed: method=GET path=/v1/client/allocation/614424f2-09f0-e64c-8f0c-cf824db519e4/stats error="Unknown allocation "614424f2-09f0-e64c-8f0c-cf824db519e4"" code=404

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.