Cherry-pick #24570 to 7.x: [Filebeat] Add Malware Bazaar to Threat Intel Module

[P1llus]

Cherry-pick of PR #24570 to 7.x branch. Original message:

Closes https://github.com/elastic/cti-team/issues/33

What does this PR do?

This PR adds the Malware Bazaar threat feed to the threat intel module of Filebeat.

Why is it important?

Malware Bazaar provides rich file metadata about malware that can assist cyber intelligence analysts, threat hunters, and incident responders during incident response and ongoing security operations.

Currently, the threat intel module for Filebeat did not have the data provided by Malware Bazaar.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Validated that ./filebeat setup imports the saved search, visualization, and dashboard
• Validated that data provided by the Threat Intel Filebeat module populates the dashboards

How to test this PR locally

This can be tested by:

cd beats/x-pack/filebeat
mage update && mage build
./filebeat modules enable threatintel
# if using Elastic Cloud, update filebeat.yml to cloud.id and cloud.auth
./filebeat setup -E setup.dashboards.directory=./build/kibana
./filebeat

Go to KIbana -> dashboards -> apply the threat intel tag

Related issues

Resolves #24569

Use cases

Threat hunting, security operations, and intelligence analysis.

Screenshots

Logs

Original data from source

  {
            "sha256_hash": "ff4ad9465d1312e24dff76c47ddb519bc46f501239e5813d4d70ac6bd7c70638",
            "sha3_384_hash": "40ce04e3e695d10b42d9ca1026476aa6076923a911652c6f263f866c8ce86e9bf2b03e1626f3ce20f6d4f1649ed3f54a",
            "sha1_hash": "eb519d0338c102103d0e8467a5dba55c99a40233",
            "md5_hash": "4f0f5e2c6bc8d65d14ff201d799824ae",
            "first_seen": "2021-03-16 16:07:32",
            "last_seen": null,
            "file_name": "tarifvertrag_chemie_bayern_entgelttabelle.js",
            "file_size": 2939,
            "file_type_mime": "text\/plain",
            "file_type": "js",
            "reporter": "dor0n1",
            "origin_country": "DE",
            "anonymous": 0,
            "signature": null,
            "imphash": null,
            "tlsh": "6E51A6E5B149E15101073734255FE86CB9B3A949D40ED420D749D7DF28B603D4E27A9E",
            "telfhash": null,
            "ssdeep": "48:M27hDQrbyxd8CROAdiQD0GMFJ0QXtAkiQKXJXWcaiMKjTlLJ\/9zhjOquc9+ac03V:5Vwbg8BxHpX4XMAF96qfmaMJ0ncTwy1w",
            "tags": [
                "gootloader"
            ],
            "code_sign": [],
            "intelligence": {
                "clamav": null,
                "downloads": "15",
                "uploads": "1",
                "mail": null
            }
        },

Data from module

{
  "_index": "filebeat-8.0.0-2021.03.16-000001",
  "_type": "_doc",
  "_id": "38de8b728560e8705f707d4607706d73f04b4d50f925608ffed7b873e66e9b67",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "c02zc1eklvdr",
      "id": "4aa831e8-28aa-4037-ab92-02cdcdf0e6ae",
      "ephemeral_id": "c6efb3b7-007f-42ad-b2bf-c34fb577a9a7",
      "type": "filebeat",
      "version": "8.0.0"
    },
    "fileset": {
      "name": "malwarebazaar"
    },
    "threatintel": {
      "malwarebazaar": {
        "intelligence": {
          "mail": {
            "CH": "low",
            "Generic": "high"
          },
          "downloads": 22,
          "uploads": 1
        },
        "tags": [
          "doc"
        ],
        "anonymous": 0,
        "code_sign": []
      },
      "indicator": {
        "geo": {
          "country_iso_code": "FR"
        },
        "first_seen": "2021-03-16T16:05:36.000Z",
        "file": {
          "extension": "doc",
          "size": 167472,
          "mime_type": "text/rtf",
          "pe": {},
          "name": "Purchase order.doc",
          "hash": {
            "sha1": "1471a4b801b065b315f130a5d512dc7597c71657",
            "sha384": "c70be406ff428593018f85cc8363635c259506197972371e24f7161011a8a46dca83ac0268ca80d8a92d6ccf93fcc9a1",
            "sha256": "623119afef4f91cf052823dd4d6b1ec9f04dc7fbe76a313035e0e4aa00888454",
            "tlsh": "30F36DA8E991CDD4CBCFC9D94A1E3A952033FA79C6C36C964438F3F50B9267F4A16850",
            "ssdeep": "3072:Cv6q8aN9Zts1unvhfKwGt7D2l8hnoo+ixxpbeBJTJzEmFhThl4O/b+8OGo3VaESF:CvkaN9Zts6vVGt/2l8nxpKJTJznhdDTP",
            "md5": "a24571eb4baea7a5b67480d524668617"
          }
        },
        "provider": "cocaman",
        "type": "file"
      }
    },
    "tags": [
      "threatintel-malwarebazaar",
      "forwarded"
    ],
    "input": {
      "type": "httpjson"
    },
    "@timestamp": "2021-03-16T16:37:32.980Z",
    "ecs": {
      "version": "1.6.0"
    },
    "related": {
      "hash": [
        "a24571eb4baea7a5b67480d524668617",
        "623119afef4f91cf052823dd4d6b1ec9f04dc7fbe76a313035e0e4aa00888454",
        "3072:Cv6q8aN9Zts1unvhfKwGt7D2l8hnoo+ixxpbeBJTJzEmFhThl4O/b+8OGo3VaESF:CvkaN9Zts6vVGt/2l8nxpKJTJznhdDTP"
      ]
    },
    "service": {
      "type": "threatintel"
    },
    "event": {
      "ingested": "2021-03-16T16:37:34.071668028Z",
      "created": "2021-03-16T16:37:32.980Z",
      "kind": "enrichment",
      "module": "threatintel",
      "category": "threat",
      "type": "indicator",
      "dataset": "threatintel.malwarebazaar"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-03-16T16:37:34.071Z"
    ],
    "@timestamp": [
      "2021-03-16T16:37:32.980Z"
    ],
    "event.created": [
      "2021-03-16T16:37:32.980Z"
    ]
  },
  "sort": [
    1615912652980
  ]
},

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b backport_24570_7.x upstream/backport_24570_7.x
git merge upstream/7.x
git push upstream backport_24570_7.x

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #25177 updated

• Start Time: 2021-04-20T18:52:24.063+0000

• Duration: 75 min 39 sec

• Commit: 15caa2e

Test stats 🧪

Test	Results
Failed	0
Passed	13576
Skipped	2285
Total	15861

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13576
Skipped	2285
Total	15861

[P1llus]

run elasticsearch-ci/docs