Feature request: Include tags when sending JSON logs

[alubbock]

Feature Request

Description

Add a talos-node or hostname field to the service and kernel logs when sending them over TCP or UDP using json_lines.

The Talos docs describe how to send logs e.g. to Loki using Fluent-bit. When logs from many nodes are sent to a centralised endpoint, there's no way to tell which node kernel and service logs came from. Adding a field with the node's name would solve this.

[smira]

This was discussed I believe couple of times, the issue here is that Talos doesn't know it's hostname until it gets actually assigned, and the hostname might change during the boot process. Any centralized logging solution has an idea of the source IP of the log sender, so it could use that to attach proper labels to the logs.

So if implemented, this would look more like "attach the hostname to the logs when known" or something like that.

[sergelogvinov]

Probably, we need to add custom key/value to json.

destinations:
    - endpoint: tcp://1.2.3.4:12345 # Where to send logs. Supported protocols are "tcp" and "udp".
      format: json_lines
      tags:
         custom_name: custom_value

[zystem]

Any centralized logging solution has an idea of the source IP of the log sender, so it could use that to attach proper labels to the logs.

You absolutely do not right.

1. There are many ways to lose the source IP address (NAT, cilium)
2. Many popular logging solutions cannot extend data with source IP at all (filebeat, fluent-bit for example)
3. Best practices in logging say that logs should be extended with source IP by the sender (syslog standard for example)
4. As for me there is no problem with adding IP instead of hostname. But there should be some field allowing to separation of logs from different nodes.

[smira]

So what we need to implement is #7356 (comment)

[zystem]

So what we need to implement is #7356 (comment)

In this case, tags should support dynamic variables. For example IP can be received from DHCP and support of environment variable can be not enough. IP tag should reflect current IP status.

[smira]

IP is not defined for Talos in a stable way - it might change, while log identifier should be static.

[alubbock]

IP is not defined for Talos in a stable way - it might change, while log identifier should be static.

Would system UUID make a good identifier for this purpose (node disambiguation)? Users can check with:
talosctl read /sys/class/dmi/id/product_uuid

[frezbo]

IP is not defined for Talos in a stable way - it might change, while log identifier should be static.

Would system UUID make a good identifier for this purpose (node disambiguation)? Users can check with: talosctl read /sys/class/dmi/id/product_uuid

not really, we have seen many systems that just have this as all zeroes

[smira]

plus Talos supports overriding UUID, it might not be available, etc.

Log sending should be robust, so we should keep it absolutely simple.