New decode_csv_fields processor

[adriansr]

This patch introduces a new processor, decode_csv_field decode_csv_fields that decodes
rows of CSV-formatted data into a string array, one element per column.

processors:
- decode_csv_fields:
    fields:
      message: csv
    separator: ,
    overwrite_keys: false
    ignore_missing: false
    trim_leading_space: false
    fail_on_error: true

[andrewkroh]

A while back I had hacked together a lookup table processor that could load data from a CSV file. I found it useful to have a setting that allowed me to directly name specific numbered columns like

- processors:
    - decode_csv_field:
        field: message
        target: "user"
        columns:
          # Target Field Name -> CSV Column Number
          email: 0 # Write column 0 to user.email.
          name:  2 # Write column 2 to user.name. (Column 1 is ignored.)

When columns is not specified then I'd have it write an array of strings to the target like you have.

[andrewkroh]

The code LGTM.

Can you add this processor to here.

And also add a changelog entry.

[adriansr]

@andrewkroh
I originally devised it using the columns mapping too, the problem is that it won't easily fit my current use-case, as I have 50+ columns and need to inspect one of the first columns (a.k.a "type") to decide which mapping to use for the rest.

So I thought maybe it was better to have this processor to decode to an array and then add a generic "extract_array" processor or do this inside an ingest pipeline. I'm planning to discuss this on today's sync with Beats team.

[andrewkroh]

This is looking good. I think it just needs a section added to the asciidocs now.

[adriansr]

@andrewkroh I've modified the processor a little bit to help align with the rest. Do you mind reviewing again? Now it has docs.

Main change is the rename to *_fields and the ability to process more than one field at the same time (I've copied the conf style from your dns processor). This adds also the new flag fail_on_error, which is also common on another processors.

[andrewkroh]

LGTM