[proofpoint_on_demand] Initial release of the Proofpoint On Demand #10472

brijesh-elastic · 2024-07-12T11:27:35Z

Proposed commit message

Create New integration package proofpoint_on_demand.

Added audit, mail and message data stream.
Added data collection logic for all the data stream.
Added the ingest pipeline for all the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for all the data stream.
Added system test cases for all the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/proofpoint_on_demand directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: proofpoint_on_demand - START ---
╭──────────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ proofpoint_on_demand │ audit       │ static    │ Verify sample_event.json │ PASS   │  123.94021ms │
│ proofpoint_on_demand │ mail        │ static    │ Verify sample_event.json │ PASS   │ 128.099553ms │
│ proofpoint_on_demand │ message     │ static    │ Verify sample_event.json │ PASS   │   145.2949ms │
╰──────────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: proofpoint_on_demand - END   ---
Done
--- Test results for package: proofpoint_on_demand - START ---
╭──────────────────────┬─────────────┬───────────┬─────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                                   │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼─────────────────────────────────────────────┼────────┼──────────────┤
│ proofpoint_on_demand │ audit       │ pipeline  │ (ingest pipeline warnings test-audit.log)   │ PASS   │ 334.507862ms │
│ proofpoint_on_demand │ audit       │ pipeline  │ test-audit.log                              │ PASS   │ 415.951639ms │
│ proofpoint_on_demand │ mail        │ pipeline  │ (ingest pipeline warnings test-mail.log)    │ PASS   │ 316.577142ms │
│ proofpoint_on_demand │ mail        │ pipeline  │ test-mail.log                               │ PASS   │ 351.665205ms │
│ proofpoint_on_demand │ message     │ pipeline  │ (ingest pipeline warnings test-message.log) │ PASS   │ 282.115325ms │
│ proofpoint_on_demand │ message     │ pipeline  │ test-message.log                            │ PASS   │ 871.319284ms │
╰──────────────────────┴─────────────┴───────────┴─────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: proofpoint_on_demand - END   ---
Done
--- Test results for package: proofpoint_on_demand - START ---
╭──────────────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                     │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ proofpoint_on_demand │             │ asset     │ dashboard proofpoint_on_demand-77feed4b-c40f-45f4-b9dd-7094a6877609 is loaded │ PASS   │      4.997µs │
│ proofpoint_on_demand │             │ asset     │ dashboard proofpoint_on_demand-ae89dee7-9dc7-4121-ba6a-93c408307ee4 is loaded │ PASS   │      1.152µs │
│ proofpoint_on_demand │             │ asset     │ dashboard proofpoint_on_demand-e84a69fa-843b-4697-8b9c-cd9b005581ef is loaded │ PASS   │      1.233µs │
│ proofpoint_on_demand │             │ asset     │ search proofpoint_on_demand-47445983-1383-4de7-9a0a-3f39f46e5b5c is loaded    │ PASS   │      1.323µs │
│ proofpoint_on_demand │             │ asset     │ search proofpoint_on_demand-7748df39-1f80-4506-8e47-afac86766d3d is loaded    │ PASS   │      1.047µs │
│ proofpoint_on_demand │             │ asset     │ search proofpoint_on_demand-f73aa7a7-3a1d-41aa-b462-308dd0fb347b is loaded    │ PASS   │      1.084µs │
│ proofpoint_on_demand │ audit       │ asset     │ index_template logs-proofpoint_on_demand.audit is loaded                      │ PASS   │        988ns │
│ proofpoint_on_demand │ audit       │ asset     │ ingest_pipeline logs-proofpoint_on_demand.audit-0.1.0 is loaded               │ PASS   │        931ns │
│ proofpoint_on_demand │ mail        │ asset     │ index_template logs-proofpoint_on_demand.mail is loaded                       │ PASS   │        986ns │
│ proofpoint_on_demand │ mail        │ asset     │ ingest_pipeline logs-proofpoint_on_demand.mail-0.1.0 is loaded                │ PASS   │        973ns │
│ proofpoint_on_demand │ message     │ asset     │ index_template logs-proofpoint_on_demand.message is loaded                    │ PASS   │      1.008µs │
│ proofpoint_on_demand │ message     │ asset     │ ingest_pipeline logs-proofpoint_on_demand.message-0.1.0 is loaded             │ PASS   │        843ns │
╰──────────────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: proofpoint_on_demand - END   ---
Done
--- Test results for package: proofpoint_on_demand - START ---
╭──────────────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ proofpoint_on_demand │ audit       │ system    │ default   │ PASS   │ 53.695690723s │
│ proofpoint_on_demand │ mail        │ system    │ default   │ PASS   │ 38.289262876s │
│ proofpoint_on_demand │ message     │ system    │ default   │ PASS   │ 37.409306523s │
╰──────────────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: proofpoint_on_demand - END   ---
Done

Related issues

Closes Proofpoint On Demand #1196

Screenshots

elasticmachine · 2024-07-16T10:08:25Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2024-07-16T10:09:39Z

@brijesh-elastic Looks like CI is failing on system tests. Can you please check?

ShourieG · 2024-07-17T11:07:45Z

Hi @brijesh-elastic, we need to add a health check to the mock service container in the docker compose and the respective /health uri path in the mock server code to resolve the system tests issue, similar to this PR.

elasticmachine · 2024-07-17T16:58:46Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/proofpoint_on_demand/data_stream/audit/agent/stream/websocket.yml.hbs

packages/proofpoint_on_demand/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

packages/proofpoint_on_demand/data_stream/audit/sample_event.json

kcreddy · 2024-07-19T06:18:05Z

packages/proofpoint_on_demand/data_stream/audit/sample_event.json

+                },
+                {
+                    "name": "login.authorization",
+                    "value": "true"


event.outcome can be derived from here.

packages/proofpoint_on_demand/data_stream/message/elasticsearch/ingest_pipeline/default.yml

packages/proofpoint_on_demand/data_stream/message/agent/stream/websocket.yml.hbs

packages/proofpoint_on_demand/data_stream/message/elasticsearch/ingest_pipeline/default.yml

packages/proofpoint_on_demand/docs/README.md

packages/proofpoint_on_demand/manifest.yml

kcreddy · 2024-07-25T05:38:34Z

packages/proofpoint_on_demand/data_stream/mail/_dev/test/pipeline/test-mail.log-expected.json

+            "email": {
+                "from": {
+                    "address": [
+                        "<mailive@example.com>"


You might need to remove characters < and > from *address and message_id

kcreddy · 2024-07-25T05:39:25Z

packages/proofpoint_on_demand/data_stream/mail/_dev/test/pipeline/test-mail.log-expected.json

+                ],
+                "id": "ZeYABCDefghN0123456OyA",
+                "kind": "event",
+                "original": "{\"pps\":{\"agent\":\"example.proofpoint.com\",\"cid\":\"mmg_abcd001\"},\"ts\":\"2023-08-17T14:54:12.949180-07:00\",\"data\":\"2017-08-17T14:54:12.949180-07:00 example sendmail[30641]: ABcDqYbx123123: to=jack@example.com, ctladdr=<user1@example.com> (8/0), delay=00:00:00, xdelay=00:00:00, mailer=*file*, tls_verify=NONE, pri=35342, dsn=2.0.0, stat=Sent\",\"sm\":{\"tls\":{\"verify\":\"NONE\"},\"stat\":\"Sent\",\"qid\":\"ABcDqYbx123123\",\"dsn\":\"2.0.0\",\"mailer\":\"*file*\",\"to\":[\"jack@example.com\"],\"ctladdr\":\"<user1@example.com> (8/0)\",\"delay\":\"00:00:00\",\"xdelay\":\"00:00:00\",\"pri\":35342},\"id\":\"ZeYABCDefghN0123456OyA\"}",


This event doesn't have a from= address, so the from could be derived from ctladdr

elasticmachine · 2024-07-25T10:32:08Z

💚 Build Succeeded

Buildkite Build
Commit: 164ec7a

History

💚 Build #13870 succeeded 79d4f12
💚 Build #13742 succeeded b833375
💔 Build #13605 failed 8f0dc7a

elastic-sonarqube · 2024-07-25T10:32:14Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-07-25T10:48:54Z

Package proofpoint_on_demand - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=proofpoint_on_demand

…lastic#10472) Create New integration package proofpoint_on_demand. Added audit, mail and message data stream. Added data collection logic for all the data stream. Added the ingest pipeline for all the data stream. Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files. Added dashboards and visualizations. Added test for pipeline for all the data stream. Added system test cases for all the data stream.

brijesh-elastic added 2 commits July 12, 2024 16:40

Initial Release

1c6efab

Update changelog entry

8f0dc7a

kcreddy added Crest Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Jul 16, 2024

kcreddy added the New Integration label Jul 16, 2024

kcreddy mentioned this pull request Jul 16, 2024

websocket: new generic integration #9926

Merged

4 tasks

ShourieG mentioned this pull request Jul 17, 2024

[filebeat][websocket] - Add retry mechanism to socket connections elastic/beats#40271

Closed

Resolve build failure

b833375

kcreddy reviewed Jul 19, 2024

View reviewed changes

Resolve comments

79d4f12

brijesh-elastic requested a review from kcreddy July 23, 2024 12:57

kcreddy reviewed Jul 25, 2024

View reviewed changes

Resolve comments

164ec7a

brijesh-elastic requested a review from kcreddy July 25, 2024 10:10

kcreddy approved these changes Jul 25, 2024

View reviewed changes

kcreddy merged commit 59b8247 into elastic:main Jul 25, 2024
5 checks passed

andrewkroh added the Integration:proofpoint_on_demand Proofpoint On Demand label Aug 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[proofpoint_on_demand] Initial release of the Proofpoint On Demand #10472

[proofpoint_on_demand] Initial release of the Proofpoint On Demand #10472

brijesh-elastic commented Jul 12, 2024

elasticmachine commented Jul 16, 2024

kcreddy commented Jul 16, 2024

ShourieG commented Jul 17, 2024 •

edited

Loading

elasticmachine commented Jul 17, 2024

kcreddy Jul 19, 2024

kcreddy Jul 25, 2024

kcreddy Jul 25, 2024

elasticmachine commented Jul 25, 2024

elastic-sonarqube bot commented Jul 25, 2024

elasticmachine commented Jul 25, 2024

[proofpoint_on_demand] Initial release of the Proofpoint On Demand #10472

[proofpoint_on_demand] Initial release of the Proofpoint On Demand #10472

Conversation

brijesh-elastic commented Jul 12, 2024

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jul 16, 2024

kcreddy commented Jul 16, 2024

ShourieG commented Jul 17, 2024 • edited Loading

elasticmachine commented Jul 17, 2024

🚀 Benchmarks report

kcreddy Jul 19, 2024

Choose a reason for hiding this comment

kcreddy Jul 25, 2024

Choose a reason for hiding this comment

kcreddy Jul 25, 2024

Choose a reason for hiding this comment

elasticmachine commented Jul 25, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Jul 25, 2024

Quality Gate passed

elasticmachine commented Jul 25, 2024

ShourieG commented Jul 17, 2024 •

edited

Loading