[Q&A]

[urupaud]

What is a problem?

Hi Team,

I’m trying to deploy fluentd as a deployment in k8s cluster exposing service as LoadBalancer (AWS Network Loadbalancer) so it can accept logs from outside the k8s cluster (windows servers) and save them in the elasticsearch cluster that is running on k8s. However though fluentd pods and loadbalancer is up and running with the healthy targets, I still can’t see any logs. I've even updated the configs to write the receiving logs to stdout to test, but still no luck. Could you please advise?

Note that I've already checked the access logs of the LoadBalancer and I can see the connections, so it seems like it's not an issue with the connectivity between the windows servers and NLB.

Thanks

Describe the configuration of Fluentd

apiVersion: v1
kind: ConfigMap
metadata:
  name: monolith-fluentd-config
data:
  fluent.conf: |-
    @include config.d/*

    <system>
      log_level debug
    </system>
    
apiVersion: v1
kind: ConfigMap
metadata:
  name: monolith-configs
data:
  nxlogs.conf: |-
    <source>
      @id nxlogs
      @type tcp
      @label @NXLOGS
      format none
      port 55446
      tag nxlogs
    </source>

    #For anything that doesnt match, print to stdout
    <label @NXLOGS>
      <match nxlogs.**>
        @type stdout
      </match>    
    </label>

Describe the logs of Fluentd

2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluentd' version '1.16.2'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-elasticsearch' version '5.4.3'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-forest' version '0.3.3'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-geoip' version '1.3.2'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-grep' version '0.3.4'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-grok-parser' version '2.6.2'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-mutate_filter' version '1.0.7'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-prometheus' version '2.1.0'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-record-reformer' version '0.9.1'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: gem 'fluent-plugin-script' version '0.1.1'
2024-04-17 03:11:16 +0000 [debug]: fluent/log.rb:341:debug: No fluent logger for internal event
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: using configuration file: <ROOT>
  <source>
    @id nxlogs
    @type tcp
    @label @NXLOGS
    format none
    port 55446
    tag "nxlogs"
    <parse>
      @type none
    </parse>
  </source>
  <label @NXLOGS>
    <match nxlogs.**>
      @type stdout
    </match>
  </label>
  <system>
    log_level debug
  </system>
</ROOT>
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: starting fluentd-1.16.2 pid=7 ruby="3.1.4"
2024-04-17 03:11:16 +0000 [info]: fluent/log.rb:362:info: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
2024-04-17 03:11:17 +0000 [info]: #0 fluent/log.rb:362:info: init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-04-17 03:11:17 +0000 [info]: fluent/log.rb:362:info: adding match in @NXLOGS pattern="nxlogs.**" type="stdout"
2024-04-17 03:11:17 +0000 [info]: fluent/log.rb:362:info: adding source type="tcp"
2024-04-17 03:11:17 +0000 [debug]: #0 fluent/log.rb:341:debug: No fluent logger for internal event
2024-04-17 03:11:17 +0000 [info]: #0 fluent/log.rb:362:info: starting fluentd worker pid=16 ppid=7 worker=0
2024-04-17 03:11:17 +0000 [info]: #0 [nxlogs] listening tcp socket bind="0.0.0.0" port=55446
2024-04-17 03:11:17 +0000 [info]: #0 fluent/log.rb:362:info: fluentd worker is now running worker=0

Environment

- Docker image (tag): fluentd:v1.16.2-debian-1.1
- Operating system: Amazon Linux 2

[daipom]

It looks like there is no problem with the Fluentd configuration.

It is possible that the data is not reaching the 55446 port on this machine, so you should check that.

It might be a good test to try manually sending TCP data from the outside using nc commands, etc.

echo "test" | nc x.x.x.x 55446 -w 1

[urupaud]

Thanks @daipom , indeed that was the case, had to update configurations in AWS NLB to make it work.