Missing Info about time

[thenovacreator]

Hey guys,

i am trying to get the time out of an nginx log into a mysql database. But it fails saying that the field "time" can not be null. Witch makes sense since this field is not nullable. But the question is ... how do i get the event time field into the mysql database? I am feeling that might be an obvious question but i can not get it working. This is how far i got so far. The first part is taken from here:

https://docs.fluentd.org/v/0.12/parser/nginx

This is the config:

<source>
        @type tail
        read_from_head true
        tag nginx
        path /var/fluentd/source/nginx/*/access.log
        pos_file /var/fluentd/data/nginx.pos
        format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
        time_format %d/%b/%Y:%H:%M:%S %z
        keep_time_key true
</source>

<match nginx>
        @type sql
        host someHost
        port 3306
        database someDB
        adapter mysql2
        username fluentd
        password somePW

        <buffer>
                flush_interval 10s
        </buffer>

        <table>
                table nginx_responses
                column_mapping 'time,method,code,path,size,referer,agent'
        </table>
</match>

Logging looks like this:

2021-08-12 17:29:45 +0200 [error]: #0 Got deterministic error again. Dump a record error_class=ActiveRecord::NotNullViolation error="Mysql2::Error: Column 'time' cannot be null" record=#<Fluent::Plugin::SQLOutput::BaseModel_1203145339::NginxResponse id: nil, time: nil, method: "POST", code: "404", path: "/url", size: "50", referer: "-", agent: "curl/7.53.1">

It would be nice if you guys could help me and also put a little hint in the description for others failing with this for hours like me.

[vpease]

try this
<filter **>
@type record_transformer
enable_ruby false

date ${time}

then, in column_mapping use date instead of time.

• It seems that format is going to the Time part of the event so it wont be available as part of the record anymore. With the filter, we are inserting the date field back to the record.