improve efficiency of Suricata processing uploaded PCAP files

[mmguero]

@mmguero cloned issue idaholab/Malcolm#325 on 2024-01-08:

Currently as uploaded PCAP files are processed, each PCAP file results in a new suricata process for that PCAP file.

This is the same behavior for Zeek and Arkime capture; however, suricata seems to have more overhead (I often notice that suricata is still running on a batch of uploaded PCAP files long after the others are done).

I came across this thread describing using suricata socket control to send PCAP files to a single long-running suricata process, then output each eve.json to a different directory per-PCAP. This would be an improvement.

[mmguero]

This is about done, although I'm getting inconsistent results into dashbaords from the runs. From the stame PCAP set over two runs:

run 1
-----
JSON lines: 32778 in 55 files
suricata start: 14:53:54
suricata end:   14:54:49
ingest start:   14:54:06
ingest end:     14:55:05

run 2
JSON lines: 33726 in 57 files
suricata start: 15:11:13
suricata end:   15:12:10
ingest start:   14:54:06
ingest end:     14:55:05

My guess is some kind of a race condition in processing the results (like the .json file isn't flushed yet or something at the point where we're copying it).

I'm wondering if rather than doing it this way, we can just create the JSON files in place rather than moving them, then let filebeat handle it. We'll need to change where the tags are coming from (from the directory name rather than the file name). I'm going to look into that.

[mmguero]

MUCH better. Still not sure why the 3 log difference :/ but looking a lot better. I still have some tests to do to make sure things recover if processes or restarted or whatever but this is about done.

run 1: 57,927 logs created, parsed and ingested in 1:13
run 2: 57,924 logs created, parsed and ingested in 1:14

[mmguero]

Here are my first performance tests, the first since I've enabled the multithreading for the suricata process, using the pcaps from the test suite and SURICATA_AUTO_ANALYZE_PCAP_THREADS=4:

new method: 57,925 logs created, parsed and ingested in 0:57 (4 threads)
old method: 57,436 logs created, parsed and ingested in 16:44 (4 threads)

So that's rougly an 18x speedup.

[mmguero]

I'm satisfied with this, marking as closed.