DiskPressure due to 80 GB /home/fluentd/core. files

[Alisher-Nabiev]

Describe the bug

hi everyone,

i got on my eks fluentd that sents log to elastiserach 7.17.

after upgrade my cluster from 1.24 to 1.25 my nodes got DiskPressure due to 80 GB /home/fluentd/core. files.

when the fluentd pod fills up the space of the node its evicted and the node in DiskPressure stat for 3-4 min after that its gone and fluentd pod runs again.

To Reproduce

--

Expected behavior

---

Your Environment

- Fluentd version: fluentd 1.16.3
- Fluent Package version: Debian GNU/Linux 12 (bookworm)
- Docker image (tag): fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
- Kernel version: 5.10.209-198.812.amzn2.x86_64

Your Configuration

@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"
@include kubernetes.conf
@include conf.d/*.conf

<filter kubernetes.var.log.containers.**.log>
@type parser
key_name log
reserve_data true
reserve_time true
hash_value_field ******-fields
remove_key_name_field false
replace_invalid_sequence false
emit_invalid_record_to_error true

@type json

<filter kubernetes.var.log.containers.****.log kubernetes.var.log.containers..log kubernetes.var.log.containers..log kubernetes.var.log.containers.**.log>
@type parser
key_name log
reserve_data true
reserve_time true
hash_value_field fields
remove_key_name_field false
replace_invalid_sequence false
emit_invalid_record_to_error true

@type json

<label @error>
<match **>
@type null

<match kubernetes.var.log.containers..log kubernetes.var.log.containers.log kubernetes.var.log.containers..log kubernetes.var.log.containers..log>
@type elasticsearch
@id out_es
@log_level debug
include_tag_key true
hosts "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
logstash_dateformat "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"
logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"
target_index_key "#{ENV['FLUENT_ELASTICSEARCH_TARGET_INDEX_KEY'] || use_nil}"
type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
include_timestamp "#{ENV['FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP'] || 'false'}"
template_name "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_NAME'] || use_nil}"
template_file "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_FILE'] || use_nil}"
template_overwrite "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_OVERWRITE'] || use_default}"
sniffer_class_name "#{ENV['FLUENT_SNIFFER_CLASS_NAME'] || 'Fluent::Plugin::ElasticsearchSimpleSniffer'}"
request_timeout "#{ENV['FLUENT_ELASTICSEARCH_REQUEST_TIMEOUT'] || '5s'}"
suppress_type_name "#{ENV['FLUENT_ELASTICSEARCH_SUPPRESS_TYPE_NAME'] || 'true'}"
enable_ilm "#{ENV['FLUENT_ELASTICSEARCH_ENABLE_ILM'] || 'false'}"
ilm_policy_id "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_ID'] || use_default}"
ilm_policy "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY'] || use_default}"
ilm_policy_overwrite "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_OVERWRITE'] || 'false'}"

flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '10M'}"
queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '64'}"
retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_forever true

Your Error Log

2024-02-28T09:46:47.341755097Z stdout F 2024-02-28 09:46:47 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-02-28T09:46:47.585579146Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2024-02-28T09:46:47.610827968Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding match in @ERROR pattern="**" type="null"
2024-02-28T09:46:47.613734496Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding filter pattern="kubernetes.var.log.containers.**unt**.log kubernetes.var.log.containers.ts**.log kubernetes.var.log.containers.g**.log kubernetes.var.log.containers.o-**.log" type="kubernetes_metadata"
2024-02-28T09:46:47.879696085Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding filter pattern="kubernetes.var.log.containers.**ed**.log" type="parser"
2024-02-28T09:46:47.963980086Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding filter pattern="kubernetes.var.log.containers.**pi**.log kubernetes.var.log.containers.ts**.log kubernetes.var.log.containers.d-**.log kubernetes.var.log.containers.o-**.log" type="parser"
2024-02-28T09:46:47.965139823Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**nt**.log kubernetes.var.log.containers.ts**.log kubernetes.var.log.containers.d-**.log kubernetes.var.log.containers.o-**.log" type="elasticsearch"
2024-02-28T09:46:47.988272344Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2024-02-28T09:46:47.988295972Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'index_name logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: logstash' doesn't have tag placeholder
2024-02-28T09:46:47.988313594Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2024-02-28T09:46:47.988316858Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'logstash_prefix k8s_test' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: k8s_test' doesn't have tag placeholder
2024-02-28T09:46:47.988387303Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2024-02-28T09:46:47.988399976Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2024-02-28T09:46:47.988420644Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2024-02-28T09:46:47.988444601Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2024-02-28T09:46:47.988451306Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2024-02-28T09:46:47.98845474Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] Need substitution: false
2024-02-28T09:46:47.988555982Z stdout F 2024-02-28 09:46:47 +0000 [debug]: #0 [out_es] 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2024-02-28T09:46:47.996009284Z stdout F 2024-02-28 09:46:47 +0000 [info]: adding source type="systemd"
2024-02-28T09:46:48.019031053Z stdout F 2024-02-28 09:46:48 +0000 [info]: adding source type="systemd"
2024-02-28T09:46:48.020042521Z stdout F 2024-02-28 09:46:48 +0000 [info]: adding source type="systemd"
2024-02-28T09:46:48.020978967Z stdout F 2024-02-28 09:46:48 +0000 [info]: adding source type="prometheus"
2024-02-28T09:46:48.044009265Z stdout F 2024-02-28 09:46:48 +0000 [info]: adding source type="prometheus_output_monitor"
2024-02-28T09:46:48.049370685Z stdout F 2024-02-28 09:46:48 +0000 [info]: adding source type="tail"
2024-02-28T09:46:48.069703396Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 starting fluentd worker pid=14176 ppid=7 worker=0
2024-02-28T09:46:48.070132396Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] buffer started instance=2460 stage_size=0 queue_size=0
2024-02-28T09:46:48.071432342Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072227564Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072557763Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072612214Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072716447Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072758772Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072815685Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.072864394Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] enqueue_thread actually running
2024-02-28T09:46:48.072956322Z stdout F 2024-02-28 09:46:48 +0000 [debug]: #0 [out_es] flush_thread actually running
2024-02-28T09:46:48.074608639Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/aws-node-ztprw_kube-system_aws-eks-nodeagent-0e2b98d8de457a0a54115b01aa9730be43ab19711aa398da798d76668fc016.log
2024-02-28T09:46:48.074766856Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/aws-node-ztprw_kube-system_aws-node-46344479e6cb1d7e7afbd81aa8cbbdcdb2d8b35f04a5580ef127c8e4025bf85.log
2024-02-28T09:46:48.074870122Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/aws-node-ztprw_kube-system_aws-vpc-cni-init-26eff4badfede5cb98984a86891cade2ee344fda2633d0f5cff5ff73f587d7.log
2024-02-28T09:46:48.074942022Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/cert-manager-cainjector-856d4df858-56lbv_cert-manager_cert-manager-7bd896c78fe5af351312c9ec613ba274d7d3b6e3a4eeb93b62c8.log
2024-02-28T09:46:48.075050646Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/datadog-4m65x_datadog_agent-04f68f1cbcd418d788882acc04c4e2c901ed513af4bb18860b04b44fe427fe6d.log
2024-02-28T09:46:48.075471828Z stdout F 2024-02-28 09:46:48 +0000 [warn]: #0 no patterns matched tag="kubernetes.var.log.containers.datadog-4m65x_datadog_agent-04f68f1cbcd418d788882acc04c4e2c901ed513af4bb18860b04b44fe427f6d.log"
2024-02-28T09:46:48.075586032Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/datadog-4m65x_datadog_init-config-74986b8824516ab49b0ca5a192232e23d994a26213fbde979617e453f05dced.log
2024-02-28T09:46:48.075701621Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/datadog-4m65x_datadog_init-volume-523849f61fabd8417a16993d7ba472a92b217fd7d787b53676e6df40.log
2024-02-28T09:46:48.075802126Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/datadog-4m65x_datadog_process-agent-87564de508f79a5f6231add87210df0c5883e6316a6ea41f166f169dc1ef45.log
2024-02-28T09:46:48.075901095Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/datadog-4m65x_datadog_trace-agent-7bb02c45216a051bc0996664a08749e3f62c74af894bcfc10d2efe418832320.log
2024-02-28T09:46:48.076014419Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/ebs-csi-node-cm8gn_kube-system_ebs-plugin-64f27aa00f9f3d4d23ddae5cf0c30b5a3a19db21dafe79cc8fce004.log
2024-02-28T09:46:48.076118294Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/ebs-csi-node-cm8gn_kube-system_liveness-probe-24291a8ac9ded233659440af1a03c509b181cf518a99e701d22c0981b54c96e.log
2024-02-28T09:46:48.076225478Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/ebs-csi-node-cm8gn_kube-system_node-driver-registrar-79cfa10bda90daafa6edebdefc9c5110a16f262c300857ebcf545624a9490ba.log
2024-02-28T09:46:48.076332847Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/efs-csi-node-lj872_kube-system_csi-driver-registrar-e755523d8f3c0426745cd0ca6b17f53f29c9df4c9aa9f6ef2a4d0be16c6cc7.log
2024-02-28T09:46:48.076519171Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/efs-csi-node-lj872_kube-system_efs-plugin-abe66087b2f98ab67d558d69fc24d76110b472a899877f633d7a55e17924da1.log
2024-02-28T09:46:48.076627941Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/efs-csi-node-lj872_kube-system_liveness-probe-5fcc7a97f55ef063adca6f2e185cc58e2221ee7488dbff1960ace0ba11794b0.log
2024-02-28T09:46:48.076732947Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/****-766f9d9745-mwp5b_external-secrets_external-secrets-839314d751c71ef5404d1e90f1c010e8eaba127833f86f6bd636246d018981d.log
2024-02-28T09:46:48.076856277Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/fluentd-kbxpg_kube-system_fluentd-ea39e7fddf8f16c5a1f31d052337cd2683aed8a932e6f078f28964e36c32cf1.log
2024-02-28T09:46:48.078365329Z stdout F 2024-02-28 09:46:48 +0000 [warn]: #0 no patterns matched tag="kubernetes.var.log.containers.fluentd-kbxpg_kube-system_fluentd-ea39e7fddf8f16c5a1f31d052337cd2683aed8a932e6f0d78f2896e36c32cf1.log"
2024-02-28T09:46:48.07848324Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/ingress-nginx-controller-85dd4d9bdc-nrbzt_ingress-nginx_controller-5f4f406c6ed7a004edd74d13d2ab2f9a457ae7d2a6114e38c70c5c6043.log
2024-02-28T09:46:48.07863442Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-proxy-r95mm_kube-system_kube-proxy-ef5ada4caf9487463a9dca7559f02aeb3d26a88f70bc24072fe5b18614dea2e.log
2024-02-28T09:46:48.078723474Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/mongodb-7c4f7b558b-4tl22_staging_mongodb-0f6fbb1a5427fb8c89bc8515721b2aaf87277faa5dfdac7b3533a7aea3bc39c.log
2024-02-28T09:46:48.078886993Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/*****-7fdc7f7c75-z9lj4_staging_myaccount-api-7e23bb01ef0abaad4b365a272681b7625e576afd426df5a28d6ca8834fc3cd0.log
2024-02-28T09:46:48.0789661Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/postgress-postgresql-0_nodered-postgresql_postgresql-59b3fc23ff2109bf61782826d9015199cd7c0b12f7fc3fe8c11583e69f84b3e.log
2024-02-28T09:46:48.079068844Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/prometheus-prometheus-node-exporter-thjf4_monitoring_node-exporter-de9c5bfea841c2ec4677a5a1552fe871c0cc32cbe88fa8f4324afcf69c3e752.log
2024-02-28T09:46:48.079142732Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/redis-master-0_staging_redis-7b5281d807f543943124f67655b14dbc883a5a2e1d7e94c0031c418a60cc7cb.log
2024-02-28T09:46:48.079231685Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/reloader-reloader-7bd5b55b85-tz4hv_default_reloader-reloader-5a22f66c4f7adfb6a7666ae0fb629cb040c40a67cbed47d6485429b001fdf1d.log
2024-02-28T09:46:48.082575753Z stdout F 2024-02-28 09:46:48 +0000 [info]: #0 fluentd worker is now running worker=0
2024-02-28T09:46:49.111343467Z stdout F 2024-02-28 09:46:49 +0000 [warn]: #0 no patterns matched tag="kubelet"
2024-02-28T09:46:49.112736483Z stderr F free(): invalid pointer
2024-02-28T09:46:49.895439811Z stdout F 2024-02-28 09:46:49 +0000 [error]: Worker 0 exited unexpectedly with signal SIGABRT

Additional context

[daipom]

@Alisher-Nabiev
With this information, we are unable to reproduce this issue and cannot judge if it is a bug or not.
If you're unsure whether it's a bug or configuration issue, please start by discussing it here (contributing guidelines).

Please adjust buffering parameters or retrying parameters and check the effect.

This could be a similar situation to this (from #4420):

https://medium.com/redbox-techblog/tuning-fluentd-retries-avoiding-duplicate-documents-in-elasticsearch-e7cb9630a453

[sajal46]

Hello, I am running fluentd daemon set on EKS 1.26 with this docker image fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch and elasticsearch 8.8.0. All the pods are up and running but its is not able to write on elasitcsearch inbetween it writes some time with this the process inside the pod keeps restarting itself giving.

log =>
dump an error event: error_class=ArgumentError error="message does not exist" location=nil tag="kubelet" time=2024-03-14 18:18:04.179411000 +0000 record={"PRIORITY"=>"6", "UID"=>"0", "GID"=>"0", "BOOT_ID"=>"00d9aaed5cdc4fdda4ee5cd677987163", "MACHINE_ID"=>"ec2200b88526bdd35cfae2fe9794a7c3", "SYSLOG_FACILITY"=>"3", "CAP_EFFECTIVE"=>"1ffffffffff", "TRANSPORT"=>"stdout", "HOSTNAME"=>"ip-10-0-2-56.ap-south-1.compute.internal", "SYSTEMD_SLICE"=>"runtime.slice", "STREAM_ID"=>"416156f818df4dd3b0942714265bdd12", "SYSLOG_IDENTIFIER"=>"kubelet", "PID"=>"3676", "COMM"=>"kubelet", "EXE"=>"/usr/bin/kubelet", "CMDLINE"=>"/usr/bin/kubelet --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime-endpoint unix:///run/containerd/containerd.sock --image-credential-provider-config /etc/eks/image-credential-provider/config.json --image-credential-provider-bin-dir /etc/eks/image-credential-provider --node-ip=10.0.2.56 --pod-infra-container-image=602401143452.dkr.ecr.ap-south-1.amazonaws.com/eks/pause:3.5 --v=2 --hostname-override=ip-10-0-2-56.ap-south-1.compute.internal --cloud-provider=external --container-runtime=remote --node-labels=eks.amazonaws.com/nodegroup-image=ami-0d08872494a6e88ae,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=charge-prod --max-pods=58", "SYSTEMD_CGROUP"=>"/runtime.slice/kubelet.service", "SYSTEMD_UNIT"=>"kubelet.service", "MESSAGE"=>"Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'"}
free(): invalid size
2024-03-15 00:40:06 +0000 [error]: Worker 0 exited unexpectedly with signal SIGABRT

then eventually the pod is recreated after sometime it starts giving error of eviction with node disk pressure. what i can i do i have already increased the disk space recrreated deamonset tried to update the fluent image but the problem persists