Breaking change in the /v1/agent/self endpoint on Consul 1.13.8

[lgfa29]

Overview of the Issue

The /v1/agent/self endpoint in Consul 1.13.8 returns the agent version with an extra line break in the end. This causes Nomad fingerprint to fail as described in hashicorp/nomad#17302.

---

Reproduction Steps

1. Start a Consul 1.13.8 agent.

   $ consul agent -dev
   ==> Starting Consul agent...
                 Version: '1.13.8
   '
              Build Date: '2023-05-15 23:52:40 +0000 UTC'
                 Node ID: 'fcc6fadf-c46d-a4be-c30e-03e0a120ac46'
               Node name: 'laoqui-GDP2M7PY2N'
              Datacenter: 'dc1' (Segment: '<all>')
                  Server: true (Bootstrap: false)
             Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
            Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
       Gossip Encryption: false
        Auto-Encrypt-TLS: false
               HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
                gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
        Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
   

2. curl /v1/agent/self and verify version returned has an extra line break.

   $ curl -s http://localhost:8500/v1/agent/self | jq '.DebugConfig.Version'
   "1.13.8\n"

Consul info for both Client and Server

Client info

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 0
build:
        prerelease =
        revision = d16b8089
        version = 1.13.8

        version_metadata =
consul:
        acl = disabled
        bootstrap = false
        known_datacenters = 1
        leader = true
        leader_addr = 127.0.0.1:8300
        server = true
raft:
        applied_index = 15
        commit_index = 15
        fsm_pending = 0
        last_contact = 0
        last_log_index = 15
        last_log_term = 2
        last_snapshot_index = 0
        last_snapshot_term = 0
        latest_configuration = [{Suffrage:Voter ID:a68a19a4-bfef-4e57-8c8a-7fa20bf123f0 Address:127.0.0.1:8300}]
        latest_configuration_index = 0
        num_peers = 0
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Leader
        term = 2
runtime:
        arch = arm64
        cpu_count = 10
        goroutines = 106
        max_procs = 10
        os = darwin
        version = go1.20.4
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 1
        event_time = 2
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 1
        members = 1
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 1
        members = 1
        query_queue = 0
        query_time = 1

[jkirschner-hashicorp]

The docs for the agent-read-self endpoint state:

The Config element contains a subset of the configuration and its format will not change in a backwards incompatible way between releases. DebugConfig contains the full runtime configuration but its format is subject to change without notice or deprecation.

If DebugConfig does not claim to be backwards-compatible/stable, we should avoid building integrations built upon it wherever possible. (I recognize that DebugConfig is a readily-available means to access information that isn't otherwise available, but it's risky for an integration.)

[david-yu]

#17520 Should resolve this issue and will be released as part of Consul 1.13.9.

[thatsk]

whats the fix for this one i just upgraded to 1.13.8 and nomad on 1.5.6 and my jobs are going in pending state. ?

[johnnyplaydrums]

Hello all! Confirming this is still on open issue right? I see all the relevant PRs are merged but am not sure how to tell when/if they are out as a release. The issue is still happening on Nomad 1.7.3 and Consul 1.17.2, I confirmed by running the agents locally (consul agent -dev and nomad agent -dev) and still see the log:

2024-01-25T10:07:04.792-0800 [WARN]  client.fingerprint_mgr.consul: unable to fingerprint consul: cluster=default attribute=consul.partition

Where should I keep an eye out for the release with the fix?

Alternatively, maybe I should open a new issue for this? I see the symptom described here is gone (extra newline has been remove), but the fingerprint warning is still being logged.

curl -s http://localhost:8500/v1/agent/self | jq '.DebugConfig.Version'
"1.17.2"

[jkirschner-hashicorp]

Hi @johnnyplaydrums: Good question. There was no changelog entry provided with the merging of the Consul PR for this (#16467), so it's not easy to tell that way. It should be included in everything Consul 1.16.0+, and was included in patch releases on Consul 1.13 - 1.15.

That fix from #16467 is definitely present in the 1.17.2 code. The git blame for that line shows it has been untouched since #16467.

That aligns with your observation that the originally reported symptom is gone. So there must be a new/different/additional reason you're receiving the fingerprint warning still.

[johnnyplaydrums]

Thanks for the quick response and explanation @jkirschner-hashicorp! I will open a new issue for this in the Nomad repo just to make sure it's being tracked.

[johnnyplaydrums]

Ah, I see it's already being tracked 🙂 hashicorp/nomad#19756. I believe the warning is just a false positive.

[johnnyplaydrums]

Final note: I recommend this issue be closed since the original bug was fixed and released. It still being open caused me some confusion when unearthing this new, unrelated (although same warning log) issue.

[jkirschner-hashicorp]

Sounds good! Thanks for the confirmation that the original symptom was resolved. Therefore, I'll close this issue for now.