mimecast: add support for v2 API endpoints

[efd6]

Proposed commit message

Migrate the message_release_logs agent to the Mimecast v2 API and add v2 API
agents for:

* archive_search_logs
* audit_events
* dlp_logs
* threat_intel_malware_customer
* threat_intel_malware_grid
* ttp_ap_logs
* ttp_ip_logs
* ttp_url_logs

The migration of message_release_logs means that this is a breaking change.

Note

The addition of siem_logs v2 support depends in a stack upgrade, so that will be done in a future PR so that users on v8.14.x can get the benefit of the v2 change for the other endpoints.

Note

Test vectors for the ttp_ip_logs were not available from the test instance. If they become available they will be added. In the meantime, the change here uses v1 data via a v2 API call structure. This is likely OK since the majority of v2 data shapes match the v1 shape.

Note

Best reviewed commit-wise.

The commits can be grouped in to families:

• common:
  • archive_search_logs
  • dlp_logs
  • message_release_logs
  • ttp_ap_logs
  • ttp_ip_logs
  • ttp_url_logs
• audit
  • audit_events
• malware:
  • threat_intel_malware_customer
  • threat_intel_malware_grid

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• For [Mimecast] Add support for API 2.0 #11159
• Not before mimecast: prevent contamination of time values with short timezone IDs #11784

Screenshots

Default state (minimised)

Mimecast v1 data streams

Mimecast v2 data streams

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0b0f5c3

History

• 💚 Build #18485 succeeded fa872c1
• 💔 Build #18473 failed 61bb221

cc @efd6

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
99.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube