dnstap-receiver crashes after running for a few days

[variverrat]

Getting the following error after running for a few days:

May 09 04:30:44 dnstap_receiver[4191249]: Task exception was never retrieved
May 09 04:30:44 dnstap_receiver[4191249]: future: <Task finished name='Task-11' coro=<cb_onconnect() done, defined at /usr/local/lib/python3.8/dist-packages/dnstap_receiver/inputs/input_socket.py:13> exceptio>
May 09 04:30:44 dnstap_receiver[4191249]: Traceback (most recent call last):
May 09 04:30:44 dnstap_receiver[4191249]:   File "/usr/local/lib/python3.8/dist-packages/dnstap_receiver/inputs/input_socket.py", line 67, in cb_onconnect
May 09 04:30:44 dnstap_receiver[4191249]:     await dnstap_decoder.cb_ondnstap(dnstap_protobuf, payload, cfg, queues_list, stats, geoip_reader, cache)
May 09 04:30:44 dnstap_receiver[4191249]:   File "/usr/local/lib/python3.8/dist-packages/dnstap_receiver/inputs/dnstap_decoder.py", line 92, in cb_ondnstap
May 09 04:30:44 dnstap_receiver[4191249]:     dns_id, dns_rcode, dns_qdcount = dns_parser.decode_dns(dns_payload)
May 09 04:30:44 dnstap_receiver[4191249]:   File "/usr/local/lib/python3.8/dist-packages/dnstap_receiver/dns/parser.py", line 26, in decode_dns
May 09 04:30:44 dnstap_receiver[4191249]:     dns_hdr = unpack_dns(data[:DNS_LEN])
May 09 04:30:44 dnstap_receiver[4191249]: struct.error: unpack requires a buffer of 12 bytes

Also noticing a large amount of memory consumed during execution. When this error appears, logging stops.

[dmachard]

can you provide your configuration ? and the type of dnstap sender ?

[variverrat]

Running unbound 1.13.1

trace:
  # log file path or null to print to stdout
  file: null
  # enable verbose mode
  verbose: false

# rest api
web-api:
  # enable or disable
  enable: false
  # web api key
  api-key: changeme
  # basicauth login
  login: admin
  # basicauth password
  password: changeme
  # listening address ipv4 0.0.0.0 or ipv6 [::]
  local-address: 0.0.0.0
  # listing on port
  local-port: 8080

# geoip support, can be used to get the country, and city
# according to the source ip in the dnstap message
geoip:
  # enable or disable
  enable: false
  # city database path in binary format
  city-database: null
  # represent country in iso mode
  country-iso: false

# dnstap receiver statistics
statistics:
  # qname lowercase
  qname-lowercase: true
  # default number of items to return in tables
  max-items: 10
  # default counters to return from api /counters
  default-counters:
   - clients
   - domains
   - query
   - response
   - qps
   - response/noerror
   - response/nxdomain
  # default top domain to return from api /tables
  default-top:
   - noerror/query
   - nxdomain/response

# read and decode dnstap messages from different sources
input:
  # read dnstap messages from tcp socket
  tcp-socket:
    # enable or disable
    enable: false
    # listening address ipv4 0.0.0.0 or ipv6 [::]
    local-address: 0.0.0.0
    # listing on port
    local-port: 6000
    # access control list
    access-control-list:
     - 0.0.0.0/0
    # enable tls on socket
    tls-support: false
    # provide certificate server path
    tls-server-cert: null
    # provide certificate key path
    tls-server-key: null

  # read dnstap messages fom unix socket
  unix-socket:
    # enable or disable
    enable: true
    # socket path
    path: /run/dnstap/dnstap.sock

  # sniff dns messages from network interface
  sniffer:
    # enable or disable
    enable: false
    # interface name to sniff
    eth-name: null
    # ip interface to sniff
    eth-ip: [ ]
    # dnstap identity
    dnstap-identity: sniffer
    # sniff on the list of dns port
    dns-port: [ 53 ]
    # incoming dns client queries
    client-query-support: true
    # outgoing dns client responses
    client-response-support: true
    # outgoing dns resolver queries
    resolver-query-support: true
    # incoming dns client responses
    resolver-response-support: true
    # forward queries
    forwarder-query-support: true
    # forward responses
    forwarder-response-support: true

  # tcp client
  tcp-client:
    # enable or disable
    enable: false
    # retry interval in seconds to connect
    retry: 1
    # remote dns server address
    remote-address: null
    # remote dns server port
    remote-port: null

# filtering feature on input, can be useful to ignore
# some dnstap messages
filter:
  # qname filtering feature with regex support
  qname-regex: null
  # dnstap identify filtering feature with regex support
  dnstap-identities: null

# forward decoded messages to ?
output:
  # forward to stdout, default output
  stdout:
    # enable or disable
    enable: false
    # format available text|json|yaml
    format: text

  # forward to log file
  file:
    # enable or disable
    enable: true
    # format available text|json|yaml
    format: json
    # log file path or null to print to stdout
    file: /var/dnstap/dnstap.log
    # max size for log file
    file-max-size: 500M
    # number of max log files
    file-count: 5

  # print metrics to stdout
  metrics:
    # enable or disable
    enable: false
    # print every N seconds
    interval: 300
    # cumulative statistics, without clearing them after printing
    cumulative: true
    # log file path or null to print to stdout
    file: null
    # max size for log file
    file-max-size: 10M
    # number of max log files
    file-count: 10

  # forward to remote tcp destination
  tcp-socket:
    # enable or disable
    enable: false
    # format available text|json|yaml
    format: text
    # delimiter
    delimiter: "\n"
    # retry interval in seconds to connect
    retry: 5
    # remote ipv4 or ipv6 address
    remote-address: null
    # remote tcp port
    remote-port: null

  # forward to syslog server
  syslog:
    # enable or disable
    enable: false
    # syslog over tcp or udp
    transport: udp
    # format available text|json
    format: text
    # retry interval in seconds to connect
    retry: 5
    # remote ipv4 or ipv6 address of the syslog server
    remote-address: null
    # remote port of the syslog server
    remote-port: null

  # forward to another remote dnstap receiver
  dnstap:
    # enable or disable
    enable: false
    # retry interval in seconds to connect
    retry: 1
    # remote ipv4 or ipv6 address of the remote dnstap receiver
    remote-address: null
    # remote port of the remote dnstap receiver
    remote-port: null
    # dnstap identity
    dnstap-identity: dnstap-receiver

[dmachard]

I added a fix to avoid the crash and add more log when the error occured.
In your case, the dns packet is missing (or too short) in the dnstap message, this is why the dnstap receiver crashes.

[variverrat]

Awesome Denis. I'll let it run for a while and see how it behaves.

[dmachard]

can you provide the log when you reproduce this issue ?

[dmachard]

no feedback, issue closed