Error "event not processed by enough 'sink' nodes" in Vault 1.16.3

[lenaing]

Describe the bug

Hello there,
We recently upgraded to 1.16.3 on June the 13th (we were on 1.15.4 before), and since started to notice multiple errors that we did not have previously:

2024-07-04T03:24:14.457+0200 [ERROR] core: failed to audit response: request_path=auth/kubernetes/path/to/auth
  error=
  | 1 error occurred:
  | \t* event not processed by enough 'sink' nodes
  |

It seems to be similar to #25549 but we are already with the #26616 fix...

Environment:

• Vault Server Version : (retrieve with vault status): 1.16.3
• Vault CLI Version (retrieve with vault version): 1.16.3
• Server Operating System/Architecture: Linux / Amd64

Vault server configuration file(s):

{
  "listener": [
    {
      "tcp": {
        "address": "0.0.0.0:8200",
        "tls_cert_file": "/path/to/config/vault.crt",
        "tls_key_file": "/path/to/config/vault.key",
        "proxy_protocol_behavior": "use_always",
        "tls_disable_client_certs": "true"
      }
    }
  ],
  "storage": {
    "postgresql": {
      "connection_url": "postgresql://user:path@db.example.com:5432/db_name?sslmode=verify-ca&connect_timeout=5",
      "table": "vault_kv_store",
      "ha_enabled": "true",
      "ha_table": "vault_ha_locks"
    }
  },
  "telemetry": {
    "prometheus_retention_time": "12h",
    "disable_hostname": true
  },
  "ui": true,
  "api_addr": "https://vault.example.com:8200",
  "pid_file": "/path/to/config/run/vault.pid",
  "plugin_directory": "/path/to/config/plugins"
}

Additional context

Our single audit device is configured as such :

{
    "type": "socket",
    "description": "Store logs in socket",
    "options": {
        "address": "/path/to/vault-audit-logs",
        "socket_type": "unix",
        "format": "json",
        "prefix": ""
    }
}

The unix socket is bound to a Syslog-NG v4.2 on the same server, with the following configuration :

@version: 4.2

# Doc: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition/3.5

options {
  flush_lines (0);        # how many lines are flushed to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch.
  time_reopen (10);       # The time to wait in seconds before a dead connection is reestablished
  log_fifo_size (10000);  # The number of messages that the output queue can store
  log_msg_size(102400000);  # Maximum length of a message in bytes. This length includes the entire message
  stats(freq(43200)); # 12h
};

...

source s_vault_audit_logs { unix-stream('/path/to/vault-audit-logs'); };
destination d_vault_audit_logs { file("/path/to/logs/audit.log" perm(0644) owner("syslogng") group("syslogng")); };
log { source(s_vault_audit_logs); destination(d_vault_audit_logs); };

I can provide more information if required,
Thanks

[miagilepner]

Hi, thanks for reporting this! An additional fix for this problem was released in Vault 1.17.1 and 1.16.5 (enterprise only). Please try one of those versions and see if that resolves your issue. If it doesn't, feel free to re-open this issue.

[luis-chan-mox]

After upgrading to vault from 1.61.1 to 1.17.2 , we have encountered the issue again .

Do anyone also have the same problem?

2024-07-23T11:37:10.556Z	|
2024-07-23T11:37:10.556Z	| \t* event not processed by enough 'sink' nodes
2024-07-23T11:37:10.556Z	| 1 error occurred:
2024-07-23T11:37:10.556Z	error= failed to audit request: path=auth/okta/verify/xxxx

[lboynton]

Yes, still seeing this on Vault 1.16.6.