agent: prevent empty server_metadata.json

[huikang]

Description

Remove os.O_TRUNC when agent opens server_metadata.json. This flag seems redundant since we will write the new timestamp to the file immediately. In case the agent crashes in-between OpenServerMetadata and WriteServerMetadata, the server_metadata.json becomes empty, leading to erroneous state where agent can't be started

An alternative way is to continue the agent if server_metadata.json is empty, but this may cause a situation where a very old agent joins a cluster.

Fix: #19720

Testing & Reproduction steps

1. Start an agent
2. Stop agent
3. Remove the content in server_metadata.json
4. Agent can't be started again

==> Log data will now stream in as it occurs:

2023-12-13T16:11:42.763-0500 [WARN]  agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
2023-12-13T16:11:42.763-0500 [WARN]  agent: bootstrap = true: do not enable unless necessary
2023-12-13T16:11:42.765-0500 [DEBUG] agent.grpc.balancer: switching server: target=consul://dc2.5ec3af79-1b29-83eb-c131-a06ecd42120d/server.dc2 from=<none> to=<none>
2023-12-13T16:11:42.768-0500 [WARN]  agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
2023-12-13T16:11:42.768-0500 [WARN]  agent.auto_config: bootstrap = true: do not enable unless necessary
2023-12-13T16:11:42.769-0500 [INFO]  agent: initialized license: id=7e3af7da-61e8-fc40-8ba9-6e14a4e30b18 expiration="2024-08-21 23:59:59.999 +0000 UTC" features="Automated Backups, Automated Upgrades, Enhanced Read Scalability, Network Segments, Redundancy Zone, Advanced Network Federation, Namespaces, SSO, Audit Logging, Admin Partitions"
2023-12-13T16:11:42.769-0500 [INFO]  agent: started routine: routine=license-manager
2023-12-13T16:11:42.769-0500 [INFO]  agent: started routine: routine=license-monitor
2023-12-13T16:11:42.769-0500 [ERROR] agent: startup error: error="error reading server metadata: unexpected end of JSON input"
^C2023-12-13T16:11:46.757-0500 [INFO]  agent: Caught: signal=interrupt
^C^C2023-12-13T16:11:52.770-0500 [ERROR] agent: startup error: error="error reading server metadata: unexpected end of JSON input"
2023-12-13T16:12:02.772-0500 [ERROR] agent: startup error: error="error reading server metadata: unexpected end of JSON input"

Links

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

[mkeeler]

LGTM

[dhiaayachi]

@huikang, a backport is missing for this PR [19935] for versions [1.15,1.16] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

[hc-github-team-consul-core]

@huikang, a backport is missing for this PR [19935] for versions [1.15,1.16] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>