One input stream to multiple output, output will affect each other.

[liguangcheng]

I configured one input and stream to 2 http outputs, when I down one of http output, after running a period of time, the normal http output will also stop, can anyone face this issue? how do I configure the fluent bit.

following are my configuration.

[INPUT]
    Name                tail
    alias               egress
    Path                /opt/fluent-bit/bin/egress/*/*/*
    Key                 event
    DB                  /opt/fluent-bit/bin/egress/fluentbit-egress.db
    Tag                 EGRESS
    Mem_Buf_Limit       20M
    Skip_Long_Lines     on
    read_from_head      true
    Exclude_Path        *.gz,*.zip

[FILTER]
    Name                    modify
    match                   EGRESS
    set     host            hostname
    set     sourcetype      egress

[OUTPUT]
    name                http
    match               EGRESS
    alias               splunk
    json_date_key       time
    host                127.0.0.1
    port                1338
    uri                 /long
    header              Authorization Splunk hehehe
    format              json
    tls                 Off
    tls.verify          False
    Retry_Limit         False

[OUTPUT]
    name                http
    match               EGRESS
    alias               kafka
    json_date_key       time
    host                127.0.0.1
    port                1339
    uri                 /long
    header              Authorization Splunk abcd1234
    format              json
    tls                 Off
    tls.verify          False
    Retry_Limit         False

[patrick-stephens]

You probably want to review https://docs.fluentbit.io/manual/administration/backpressure

Also note that you're configuring infinite retries not disabling them: https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configuring-retries - you want no_retries for that.

[liguangcheng]

Event if introduce function in_tail_progress_check_callback to remedies, fluent bit still faces the log may missing due to high speed write while low speed stream to output once write stop after a while.

[patrick-stephens]

I would use the latest version, nothing will be updated for 2.0.8 only new versions.

[liguangcheng]

Yes, I will upgrade the fluent bit later, but I am worried that the latest version will not be able to solve this problem from the root.

[liguangcheng]

@patrick-stephens sorry to disturb you again, following is part of my configuration, I define two pipelines for same path, that means two independent INPUT to monitor same log directory, and according to different tag the same log will be route to different backend system like http and Kafka, this works but I face a serious problem that is consume huge memory, while only one input just only consume little memory, can you please help to analysis, than you very much

the reason why I want to define two pipelines for one directory is in the case of only one input, if one of the two backends stops, the other will also be affected.

[INPUT]
    name                tail
    tag                 event.kafka.ingress
    alias               kafka.ingress
    buffer_chunk_size   32k
    buffer_max_size     256k
    read_from_head      true
    refresh_interval    5
    rotate_wait         10
    skip_empty_lines    off
    skip_long_lines     true
    key                 message
    db                  /var/log/logshipper/kafka.ingress.db
    db.sync             normal
    db.locking          true
    db.journal_mode     off
    path                /var/log/ses/ingress/*/*/*/*,/var/log/ses/ingress/*/*/*/*/*,/var/log/ses/ingress/*/*/*/*/*/*
    exclude_path        /var/log/logshipper/logshipper.log,/var/log/ses/ingress/*.gz,/var/log/ses/ingress/*.tgz
    mem_buf_limit       20MB
    parser              json
    ignore_older        11m

[INPUT]
    name                tail
    tag                 event.http.ingress
    alias               http.ingress
    buffer_chunk_size   32k
    buffer_max_size     256k
    read_from_head      true
    refresh_interval    5
    rotate_wait         10
    skip_empty_lines    off
    skip_long_lines     true
    key                 message
    db                  /var/log/logshipper/ingress.db
    db.sync             normal
    db.locking          true
    db.journal_mode     off
    path                /var/log/ses/ingress/*/*/*/*,/var/log/ses/ingress/*/*/*/*/*,/var/log/ses/ingress/*/*/*/*/*/*
    path_key            fluentbitLogFilePath
    exclude_path        /var/log/logshipper/logshipper.log,/var/log/ses/ingress/*.gz,/var/log/ses/ingress/*.tgz
    mem_buf_limit       20MB
    parser              json
    ignore_older        11m

[FILTER]
    name                nest
    match               event.kafka.ingress
    operation           nest
    wildcard            *
    nest_under          message
[FILTER]
    name                modify
    match               event.kafka.ingress
    set host            ${POD_NAME}
    set messageType     ingress

[FILTER]
    name                nest
    match               event.http.ingress
    operation           nest
    wildcard            *
    nest_under          message
[FILTER]
    name                modify
    match               event.http.ingress
    set msgtype         ingress
    set host            ${POD_NAME}
    set index           dfsIndex
[FILTER]
    name                nest
    match               event.http.ingress
    operation           nest
    wildcard            message
    wildcard            msgtype
    nest_under          event

[OUTPUT]
    name                kafka
    match               event.kafka.ingress
    alias               kafka.ingress
    format              json
    brokers             lgc-kafka.stc-n41-lsv4.svc.cluster.local:9092
    topics              lgc-events
    timestamp_key       time
    rdkafka.compression.codec                        gzip
    rdkafka.client.id                                ${POD_NAME}
    rdkafka.debug                                    broker,topic,consumer
    rdkafka.message.max.bytes                        1280000
    rdkafka.request.required.acks                    1
    rdkafka.queue.buffering.max.messages             10000

[OUTPUT]
    name                http
    match               event.http.ingress
    alias               http.ingress
    json_date_key       time
    json_date_format    epoch
    host                lsv-test-helm-eric-ses-backend-simulator.stc-n41-lsv
    uri                 /services/collector/event
    port                2222
    retry_limit         false
    log_response_payload false
    header              Authorization Splunk abcd1234
    format              json
    tls                 off
    tls.verify          false

[liguangcheng]

To add, actually in my environment, I have ingress, egress, tel, event 4 type log, so that means when I fully configured then there are 8 pipelines in fluent bit, the above configuration just a example that show ingress how to configure.

When I just configure 4 pipelines for example, just route logs to http backend system, the memory consumption is relative small(500M/process), but when I fully configure 8 pipelines then the memory consumption will really high(7.5G/process).