How can I use fluentd log collection on Bottlerocket?

[derekwlmscfa]

What I'd like:

Support fluentd log collection on Bottlerocket nodes. Provide configs/manifests, examples, and/or documentation.

Ideally, support both Bottlerocket and standard (Amazon Linux 2) AMIs from the same manifests, including a shared fluentd ConfigMap.

Background

We use fluentd (with the cloudwatch_logs plugin) to forward logs to AWS CloudWatch Logs. We have clusters with mixed node types (Bottlerocket and standard AMIs).

A valid fluentd configuration for our standard nodes (docker runtime, Amazon LInux 2) does not work on Bottlerocket nodes (containerd, SELinux). Logs are not forwarded, and fluentd logs on Bottlerocket nodes contain a variety of errors (like "pattern not matched"), depending on the config.

Supporting this will likely require updated fluentd configs (different log paths, formats) and possibly securityContext.

See also issue #850.

Any alternatives you've considered:

1. If "out of the box" working configs aren't available, then documented examples and guidance would help in creating my own

2. Add api services to examine files on the node

   Developing fluentd configs is complicated on Bottlerocket nodes due to its restrictions. For standard nodes, I can use SSM to get to the node, examine logs in /var/log/containers, and make the fluentd config changes. For Bottlerocket nodes, I can only SSM to the control container. I can start the admin container, but our security restrictions prohibit direct ssh (and certainly prohibit public IPs). Perhaps if I could use the control container's apiclient to examine files, I could work without a control container connection

[backjo]

I wonder if this just needs a CRI parser enabled on the FluentD configuration to work. We had the same issue with FluentBit not picking up logs until we enabled the CRI parser for BottleRocket nodes.

[derekwlmscfa]

I wonder if this just needs a CRI parser enabled on the FluentD configuration to work. We had the same issue with FluentBit not picking up logs until we enabled the CRI parser for BottleRocket nodes.

Thanks, @backjo, that does look promising. I finally got a chance to check the node logs (I had to create a new cluster on my own account to ssh into the admin container and run sheltie). The formats I'm seeing look like CRI:

bash-5.0# cd /var/log/containers
bash-5.0# ls
aws-node-n4tkt_kube-system_aws-node-cc557f36f696a6ba898019d18f87682b4cf56b283f30e8afd1c6829678075a7f.log
aws-node-n4tkt_kube-system_aws-vpc-cni-init-46cb87d5eabfae8890a0e04eb723c66f22b6206307fb09f42853029949b008e2.log
coredns-75b44cb5b4-9frnj_kube-system_coredns-1b0816b0a267685344fe48a55ccf45d5410fc77e2bdffc2e4ba7fdfdf065d362.log
coredns-75b44cb5b4-jktvq_kube-system_coredns-29095aadefeb93c8dfe30c17d8cce2c4681940608371637b2b32366ec3bbddc9.log
kube-proxy-t8j8t_kube-system_kube-proxy-fc72918876a7094585ae1515fa7bfd81bc6cc32f4e83cb1cf78e11a017da38a1.log
bash-5.0# tail aws-node-n4tkt_kube-system_aws-vpc-cni-init-46cb87d5eabfae8890a0e04eb723c66f22b6206307fb09f42853029949b008e2.log 
2021-03-16T13:48:12.537813513Z stderr F + PRIMARY_IF=eth0
2021-03-16T13:48:12.537824676Z stderr F + sysctl -w net.ipv4.conf.eth0.rp_filter=2
2021-03-16T13:48:12.539482853Z stdout F net.ipv4.conf.eth0.rp_filter = 2
2021-03-16T13:48:12.539920166Z stderr F + cat /proc/sys/net/ipv4/conf/eth0/rp_filter
2021-03-16T13:48:12.540290611Z stdout F 2
2021-03-16T13:48:12.540577337Z stderr F + '[' false == true ']'
2021-03-16T13:48:12.540587445Z stderr F + sysctl -e -w net.ipv4.tcp_early_demux=1
2021-03-16T13:48:12.541883687Z stdout F net.ipv4.tcp_early_demux = 1
2021-03-16T13:48:12.542582823Z stderr F + echo 'CNI init container done'
2021-03-16T13:48:12.542596315Z stdout F CNI init container done

bash-5.0# tail coredns-75b44cb5b4-9frnj_kube-system_coredns-1b0816b0a267685344fe48a55ccf45d5410fc77e2bdffc2e4ba7fdfdf065d362.log 
2021-03-16T13:48:30.749106186Z stdout F .:53
2021-03-16T13:48:30.749151666Z stdout F [INFO] plugin/reload: Running configuration MD5 = 4e6d221869b89618ae45c73c146779f5
2021-03-16T13:48:30.749156544Z stdout F CoreDNS-1.6.6
2021-03-16T13:48:30.749161265Z stdout F linux/amd64, go1.13.12, 6a7a75e0

Which fluentbit parser are you using? Is it this, this, or something else?

[springroll12]

@backjo Can you elaborate on what you needed to do to get fluent-bit working?

[backjo]

@springroll12 we use the following for collecting container logs. It uses the default CRI parser:

                            [INPUT]
                                Name              tail
                                Tag               kube.*
                                Path              /var/log/containers/*.log
                                Parser            cri
                                DB                /var/log/flb_kube.db
                                Mem_Buf_Limit     5MB
                                Skip_Long_Lines   On
                                Refresh_Interval  10
                                

[derekwlmscfa]

Thanks for the suggestions, @backjo and others! I was able to get good results with fluentd using:

• The cri parser, when using only Bottlerocket nodes (containerd runtime only), with source containing:

        <parse>
          @type cri
        </parse>

• The multi-format parser when using mixed nodes (Bottlerocket and Amazon Linux), as follows:

        <parse>
          @type multi_format
          <pattern>
            format json
            time_format %Y-%m-%dT%H:%M:%S.%NZ
          </pattern>
          <pattern>
            format regexp
            expression /^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z) (?<output>\w+) (?<partial_flag>[FP]) (?<message>.+)$/
          </pattern>
        </parse>

I added some further config for our unique log req'ts, but that's the gist.

Turns out, this was fairly straightforward, so unless there's further interest, I'll close the issue.

[jhaynes]

Glad you were able to get this resolved. @backjo Thanks for the help! I'll close this for now, but feel free to re-open or cut a new issue if you're still running into problems.