Guidance Needed: Converting Logstash Config with Nested Conditionals and Handling Multiple Input Files in Vector

[JeeIn-Shin]

Hi Vector Community,

Our team is migrating from Logstash to Vector, and I’m currently focused on converting the output.conf from Logstash's pipeline-parser. However, I’m facing challenges with nested conditionals and handling multiple input files during the migration.

Here’s an example of the Logstash configuration we’re working with:

logstash pipeline-parser/output.conf

if "__###__" in [tags] or
    ( [input_type] == "log" ) or
    [input_type] == "es" {
    ...
} else if {
    ...
    if [src] or [dst] {
        ruby {
            init => "
                @file_path = '/usr/share/user.csv'
            "
        ...
        }
    }
    if [src] {
        translate {
            source => "[src]"
            target => "[tmp]"
            dictionary_path => "/usr/share/indict.csv"
            refresh_interval => 300
        }
    }
}

We’ve decided to use multiple config files and are dividing each pipeline into sources, transforms, and sinks. While we’ve successfully converted the sources and sinks, we’re currently stuck at the transforms phase, particularly due to file read/write operations and handling nested conditionals.

Specific Questions:
Nested conditionals: What’s the best way to handle deeply nested if structures in Vector’s configuration format?
File operations in transforms: In Logstash, we use filters like Ruby and Translate to read external files (e.g., /usr/share/user.csv, /usr/share/indict.csv). Vector’s remap transform doesn’t seem to support direct file reads. Is there a recommended workaround?
Additionally, we are specifying multiple files as input in sources like this:

[sources.my_source_id]
type = "file"
include = [ "/tmp/foo.csv", "/tmp/bar.proto" ]

In the transforms stage, how can we distinguish which log event originated from which file? For example:

Is there metadata (e.g., file name or path) available that identifies the source file?
How can this metadata be accessed in a remap transform to process events differently based on their input file?
I’d greatly appreciate any guidance, examples, or resources to help resolve these issues.

Thank you for your support!

[pront]

Hi @JeeIn-Shin,

I will try to answer as much as I can with the context I have but feel free to follow up!

---

File operations in transforms: In Logstash, we use filters like Ruby and Translate to read external files (e.g., /usr/share/user.csv, /usr/share/indict.csv). Vector’s remap transform doesn’t seem to support direct file reads. Is there a recommended workaround?

Remap/VRL avoids this by design. You will have to use the file source to read files. You might also be interested in enrichment tables:

• https://vector.dev/guides/level-up/csv-enrichment-guide/
• https://vector.dev/docs/reference/glossary/#enrichment-tables

---

In the transforms stage, how can we distinguish which log event originated from which file?

The events coming out of the file source will have a .file field on them. See example here. Then can you use that in the following components e.g. in a Remap transform:

[transform.example_remap]
type = "remap"
inputs = ["my_source"]
source = '''
  if .file == "/tmp/foo.csv" {
    .some_field = 0
  } else if .file == "/tmp/bar.csv" {
    .some_field = 1
  }
'''

---

Nested conditionals: What’s the best way to handle deeply nested if structures in Vector’s configuration format?

The goal here is not clear to me. Are you asking if it's possible to conditionally hook up components in a pipeline? I don't think we support that.

You can have nested conditionals in Remap with VRL:

if .file == "a" {
    0
} else if file == "b" {
    1
} else if file == "c" {
    2
} else {
   4
}

The above will work on events that are coming in from the inputs.