[Filebeat] Add Malware Bazaar to Threat Intel Module

[peasead]

Closes https://github.com/elastic/elastic-security-labs/issues/61

What does this PR do?

This PR adds the Malware Bazaar threat feed to the threat intel module of Filebeat.

Why is it important?

Malware Bazaar provides rich file metadata about malware that can assist cyber intelligence analysts, threat hunters, and incident responders during incident response and ongoing security operations.

Currently, the threat intel module for Filebeat did not have the data provided by Malware Bazaar.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Validated that ./filebeat setup imports the saved search, visualization, and dashboard
• Validated that data provided by the Threat Intel Filebeat module populates the dashboards

How to test this PR locally

This can be tested by:

cd beats/x-pack/filebeat
mage update && mage build
./filebeat modules enable threatintel
# if using Elastic Cloud, update filebeat.yml to cloud.id and cloud.auth
./filebeat setup -E setup.dashboards.directory=./build/kibana
./filebeat

Go to KIbana -> dashboards -> apply the threat intel tag

Related issues

Resolves #24569

Use cases

Threat hunting, security operations, and intelligence analysis.

Screenshots

Logs

Original data from source

  {
            "sha256_hash": "ff4ad9465d1312e24dff76c47ddb519bc46f501239e5813d4d70ac6bd7c70638",
            "sha3_384_hash": "40ce04e3e695d10b42d9ca1026476aa6076923a911652c6f263f866c8ce86e9bf2b03e1626f3ce20f6d4f1649ed3f54a",
            "sha1_hash": "eb519d0338c102103d0e8467a5dba55c99a40233",
            "md5_hash": "4f0f5e2c6bc8d65d14ff201d799824ae",
            "first_seen": "2021-03-16 16:07:32",
            "last_seen": null,
            "file_name": "tarifvertrag_chemie_bayern_entgelttabelle.js",
            "file_size": 2939,
            "file_type_mime": "text\/plain",
            "file_type": "js",
            "reporter": "dor0n1",
            "origin_country": "DE",
            "anonymous": 0,
            "signature": null,
            "imphash": null,
            "tlsh": "6E51A6E5B149E15101073734255FE86CB9B3A949D40ED420D749D7DF28B603D4E27A9E",
            "telfhash": null,
            "ssdeep": "48:M27hDQrbyxd8CROAdiQD0GMFJ0QXtAkiQKXJXWcaiMKjTlLJ\/9zhjOquc9+ac03V:5Vwbg8BxHpX4XMAF96qfmaMJ0ncTwy1w",
            "tags": [
                "gootloader"
            ],
            "code_sign": [],
            "intelligence": {
                "clamav": null,
                "downloads": "15",
                "uploads": "1",
                "mail": null
            }
        },

Data from module

{
  "_index": "filebeat-8.0.0-2021.03.16-000001",
  "_type": "_doc",
  "_id": "38de8b728560e8705f707d4607706d73f04b4d50f925608ffed7b873e66e9b67",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "c02zc1eklvdr",
      "id": "4aa831e8-28aa-4037-ab92-02cdcdf0e6ae",
      "ephemeral_id": "c6efb3b7-007f-42ad-b2bf-c34fb577a9a7",
      "type": "filebeat",
      "version": "8.0.0"
    },
    "fileset": {
      "name": "malwarebazaar"
    },
    "threatintel": {
      "malwarebazaar": {
        "intelligence": {
          "mail": {
            "CH": "low",
            "Generic": "high"
          },
          "downloads": 22,
          "uploads": 1
        },
        "tags": [
          "doc"
        ],
        "anonymous": 0,
        "code_sign": []
      },
      "indicator": {
        "geo": {
          "country_iso_code": "FR"
        },
        "first_seen": "2021-03-16T16:05:36.000Z",
        "file": {
          "extension": "doc",
          "size": 167472,
          "mime_type": "text/rtf",
          "pe": {},
          "name": "Purchase order.doc",
          "hash": {
            "sha1": "1471a4b801b065b315f130a5d512dc7597c71657",
            "sha384": "c70be406ff428593018f85cc8363635c259506197972371e24f7161011a8a46dca83ac0268ca80d8a92d6ccf93fcc9a1",
            "sha256": "623119afef4f91cf052823dd4d6b1ec9f04dc7fbe76a313035e0e4aa00888454",
            "tlsh": "30F36DA8E991CDD4CBCFC9D94A1E3A952033FA79C6C36C964438F3F50B9267F4A16850",
            "ssdeep": "3072:Cv6q8aN9Zts1unvhfKwGt7D2l8hnoo+ixxpbeBJTJzEmFhThl4O/b+8OGo3VaESF:CvkaN9Zts6vVGt/2l8nxpKJTJznhdDTP",
            "md5": "a24571eb4baea7a5b67480d524668617"
          }
        },
        "provider": "cocaman",
        "type": "file"
      }
    },
    "tags": [
      "threatintel-malwarebazaar",
      "forwarded"
    ],
    "input": {
      "type": "httpjson"
    },
    "@timestamp": "2021-03-16T16:37:32.980Z",
    "ecs": {
      "version": "1.6.0"
    },
    "related": {
      "hash": [
        "a24571eb4baea7a5b67480d524668617",
        "623119afef4f91cf052823dd4d6b1ec9f04dc7fbe76a313035e0e4aa00888454",
        "3072:Cv6q8aN9Zts1unvhfKwGt7D2l8hnoo+ixxpbeBJTJzEmFhThl4O/b+8OGo3VaESF:CvkaN9Zts6vVGt/2l8nxpKJTJznhdDTP"
      ]
    },
    "service": {
      "type": "threatintel"
    },
    "event": {
      "ingested": "2021-03-16T16:37:34.071668028Z",
      "created": "2021-03-16T16:37:32.980Z",
      "kind": "enrichment",
      "module": "threatintel",
      "category": "threat",
      "type": "indicator",
      "dataset": "threatintel.malwarebazaar"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-03-16T16:37:34.071Z"
    ],
    "@timestamp": [
      "2021-03-16T16:37:32.980Z"
    ],
    "event.created": [
      "2021-03-16T16:37:32.980Z"
    ]
  },
  "sort": [
    1615912652980
  ]
},

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24570 updated

• Start Time: 2021-04-20T13:25:45.062+0000

• Duration: 56 min 29 sec

• Commit: d68e9ff

Test stats 🧪

Test	Results
Failed	0
Passed	13610
Skipped	2271
Total	15881

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13610
Skipped	2271
Total	15881

[peasead]

Close, not sure of the pr-merge or filebeat-build tests needed.

[P1llus]

You would need to add the fileset to the timestamp ignore list, That's found here: https://github.com/elastic/beats/blob/master/filebeat/tests/system/test_modules.py#L276

[P1llus]

Approved for now after the changes from Andrew K is in and the build passes. Any small updates we might want to do would need to be added after FF.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-ti-malware-bazaar upstream/filebeat-ti-malware-bazaar
git merge upstream/master
git push upstream filebeat-ti-malware-bazaar