20210302 events queues stores

[np5]

Update the event pipeline.

• Better AWS SNS/SQS queues with multiple SNS publisher threads, and support for batch consumers
• Add bulk_store method on the Elasticsearch store to speed up the indexing of the events
• Change the machine and probe events store API to increase the compatibility with the stores. For example, the pagination of events does not make really sense, especially going back to the previous page of more recent events.
• More granular use of the different store capabilities. Some stores do not implement some interfaces, and it is reflected in the UI. Zentral can now run with a principal store only offering links to its own UI for example.
• BREAKING CHANGE The probes matching an event are now serialized in that event. This way, the probe events can be fetched from the stores, even if the filtering done in the probes cannot be reproduced in the search queries (no full text and full object indexing). The events are simply fetched using the probe ID. But that also means that inactive probes cannot be used anymore to look at the past events, because those events in the stores do not contain a reference to these probes. The probe events view will only show events that were a match when the probe was active.
• The machine event metadata is serialized during the event enrichment. The process and store pipeline steps can now run without querying the database.

[np5]

WARNING the new code used to only remove an event from a SQS queue if the generated events have been successfully published is maybe too much, and needs to be tested at scale. It could be removed, if loosing some events in case a SQS queue or SNS topic is unavailable is acceptable.