Support VPC Flow Logs v6 fields for TGW

[bbayles]

This PR is an alternative to #58.

I've added support for each of the version 6 fields shown here. This should cause flowlogs_reader to be able to interpret logs with Transit Gateway records.

[tejasnadgoud]

The logs we have do not necessarily abide by these aws tgw headers. The case we have is a one of a kind where tgw logs are mixed with other vpc flowlogs. For example: the value for account-id is set to TransitGateway . In ideal case it would be the resource-type that should have been set to that value.

[bbayles]

The library should support mixed logs.

This PR should allow the user to do:

all_flow_logs = S3FlowLogsReader(...)
no_tgw_logs = (x for x in all_flow_logs if x.account_id != 'TransitGateway')

This would filter out the TGW logs.

[BThacker]

I think this will help as well.

The fields are mixmatched across the board.

Here is an example of one of the VPCFlowLogs s3 files, that shows how they are mixmatched when a Transit gateway logs are injected in them. The fields not aligning is an issue for much more than just account_id and the default fields for TGW seem to be much wider. No idea why AWS seems to allow the logs to contain both.

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2 999999999999 eni-99999999999999999 10.10.10.10 10.10.10.11 65532 389 17 1 84 9999999999 9999999999 ACCEPT OK
6 TransitGateway 999999999999 tgw-99999999999999999 tgw-attach-99999999999999999 999999999999 999999999999 - vpc-99999999999999999 - subnet-99999999999999999 - eni-99999999999999999 usw2-az4 usw2-az3 tgw-attach-99999999999999999 10.10.10.10 10.10.10.11 9769 123456 6 7 1234 9999999999 8888888888 OK IPv4 0 0 0 0 - us-west-2 ingress - -

[bbayles]

Maybe there are two different things writing logs to the same location? And they have different formats configured?

I've added a check_column_count keyword to filter out rows with the wrong number of columns.

[BThacker]

Maybe there are two different things writing logs to the same location? And they have different formats configured?

I've added a check_column_count keyword to filter out rows with the wrong number of columns.

That is what seems to happen. If you turn on TGW logging and point it to the same S3 bucket, it will combine the logs and you end up with this result.

[BThacker]

I believe this should default to True, no? This would prevent anything currently relying on this from needing to instantiate the class differently to take advantage of the added functionality. I can't think of a scenario in which we wouldn't want to check the column count (although future scenarios may arise so I like the added option of a toggle), just think it should be True. Thoughts?

[bbayles]

I set it to False by default because it means we have to process every row twice, and because it's just a heuristic - as the comment notes, having correct number of columns isn't the same as having the same columns as expected.

[BThacker]

@bbayles after some more discussion internally we've decided to drop the desire to check the column count and loop over each record twice. It turns out this is an edge case created specifically for one scenario and not something that can commonly happen by sending flow logs and transit gateway logs to S3. However we would still like to keep your changes for the future handling of TGW records. Can you remove the check column count and just leave the extra slots? If so we can approve and roll this up.

Our new direction for handling this specific case will just be to skip over the record if it valueerrors and continuing parsing the rest of the file. See https://github.com/obsrvbl/flowlogs-reader/tree/try_catch_valueerror

[bbayles]

I've reverted the last two commits - thanks.