Add packet attribute triage function

- Add triage functionality for scoring with attributes of each protocol. - Add the `target_attribute` to the `Match` trait. - Implement `score_by_packet_attr` under `Match` trait. - Modify the `ValueKind` enum to support different types of input. - Remove the `tor` module file. The structures (`HttpEventFields`, `TorConnection`) and implementations within that module have been moved to `crate::event::http`. - Fix HTTP detection events to consistently use `referrer` instead of `referrer` and `referer` interchangeably. - Change the type of fields in the detection event structure for some protocols. - `post_body`: `Vec<u8>` to `String`. - `chaddr`: `Vec<u8>` to `String`. - `class_id`: `Vec<u8>` to `String`. - `client_id`: `Vec<u8>` to `String`. Close: #354
petabi · Dec 18, 2024 · 829f5a4 · 829f5a4
1 parent 61daf7b
commit 829f5a4
Show file tree

Hide file tree

Showing 27 changed files with 1,498 additions and 195 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,35 @@ file is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and
 this project adheres to [Semantic
 Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- Added triage functionality for scoring with attributes of each raw event.
+  - Added a new enum type `AttrValue`. This type is used to convert the
+    attribute value of each raw event to its corresponding type to perform
+    comparison operations.
+  - Added the `target_attribute` to the `Match` trait to generate an `AttrValue`
+    from the field in all detection event.
+  - Implemented `score_by_packet_attr` under `Match` trait.
+
+### Changed
+
+- Removed the `tor` module file. The structures (`HttpEventFields`,
+  `TorConnection`) and implementations within that module have been moved to
+  `crate::event::http`.
+- Fixed HTTP detection events to consistently use `referrer` instead of
+  `referrer` and `referrer` interchangeably.
+- Modified the `ValueKind` enum to support different types of input for packet
+  attribute triage.
+- Changed the type of fields in the detection event structure for some raw
+  event. This change allows users to see meaningful values directly without
+  having to do any special conversion for that field.
+  - `post_body`(HTTP): `Vec<u8>` to `String`.
+  - `chaddr`(BOOTP): `Vec<u8>` to `String`.
+  - `class_id`(DHCP): `Vec<u8>` to `String`.
+  - `client_id`(DHCP): `Vec<u8>` to `String`.
+
 ## [0.33.0] - 2024-12-12
 
 ### Changed
@@ -327,7 +356,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Changed the return type of `Store::network_map` to `IndexedTable<Network>`
   to enhance security by preventing direct exposure of internal structure.
 - Replaced `IndexedMap::get_by_id` function with `Indexed::get_by_id`, providing
-   a more structured and type-safe result.
+  a more structured and type-safe result.
   - Previously, the function returned a binary representation of the key-value
     pair: `Result<(Option<impl AsRef<[u8]>>, Option<impl AsRef<[u8]>>)>`.
   - Now, it returns `Result<Option<T>>`, where T is the entry type.
@@ -343,8 +372,8 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   more straightforward and human-readable format compared to the raw binary
   format exposed by `IndexSet`.
 - Replaced the `IndexedTable<Category>::get`, `IndexedTable<Qualifier>::get` and
- `IndexedTable<Status>::get` method with the more general function
- `IndexedTable<R>::get_by_id`. This change enhances flexibility by allowing
+  `IndexedTable<Status>::get` method with the more general function
+  `IndexedTable<R>::get_by_id`. This change enhances flexibility by allowing
   retrieval based on any type R rather than being limited to a specific category.
   Existing code using get for categories should be updated to use get_by_id with
   the appropriate type.
@@ -386,7 +415,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Added new functions to facilitate insert, remove, and update operations,
   ensuring a more controlled and secure template management.
 - Introduced `Structured`, `Unstructured`, `StructuredClusteringAlgorithm` and
- `UnstructuredClusteringAlgorithm` to describe data stored in `Table<Template>`.
+  `UnstructuredClusteringAlgorithm` to describe data stored in `Table<Template>`.
 - Introduced `TriageResponse` to describe data stored in `IndexedTable<TriageResponse>`.
 - Introduced `TriageResponseUpdate` to support `TriageResponse` record update.
 - Added new functions to facilitate insert, remove, and update operations,
@@ -607,7 +636,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Removed `batch_info` and `scores` arguments from `Model::from_storage` function.
   These arguments were previously used for custom initialization of the
   `batch_info` and `scores` fields within the model. This change means that when
-  you create a model using `Model::from_storage`, the  `batch_info` and `scores`
+  you create a model using `Model::from_storage`, the `batch_info` and `scores`
   fields will now be initialized with their default values. If you previously
   relied on custom values for these fields, you will need to update your code accordingly.
 
@@ -740,6 +769,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Modified `FtpBruteForce` by adding an `is_internal` field which is a boolean
   indicating whether it is internal or not.
 
+[Unreleased]: https://github.com/aicers/review-database/compare/0.33.0...main
 [0.33.0]: https://github.com/petabi/review-database/compare/0.32.0...0.33.0
 [0.32.0]: https://github.com/petabi/review-database/compare/0.31.0...0.32.0
 [0.31.0]: https://github.com/petabi/review-database/compare/0.30.0...0.31.0

diff --git a/Cargo.toml b/Cargo.toml
@@ -52,6 +52,7 @@ tokio = { version = "1", features = ["macros"] }
 tokio-postgres-rustls = "0.13"
 toml = "0.8.13"
 tracing = "0.1"
+attrievent = { git = "https://github.com/aicers/attrievent.git", rev = "c10065a" }
 
 [dev-dependencies]
 tempfile = "3"

diff --git a/src/event.rs b/src/event.rs
@@ -20,7 +20,6 @@ mod smtp;
 mod ssh;
 mod sysmon;
 mod tls;
-mod tor;
 
 use std::{
     collections::HashMap,
@@ -57,8 +56,9 @@ pub use self::{
     },
     ftp::{BlockListFtp, FtpBruteForce, FtpBruteForceFields, FtpEventFields, FtpPlainText},
     http::{
-        BlockListHttp, BlockListHttpFields, DgaFields, DomainGenerationAlgorithm, HttpThreat,
-        HttpThreatFields, NonBrowser, RepeatedHttpSessions, RepeatedHttpSessionsFields,
+        BlockListHttp, BlockListHttpFields, DgaFields, DomainGenerationAlgorithm, HttpEventFields,
+        HttpThreat, HttpThreatFields, NonBrowser, RepeatedHttpSessions, RepeatedHttpSessionsFields,
+        TorConnection,
     },
     kerberos::{BlockListKerberos, BlockListKerberosFields},
     ldap::{BlockListLdap, LdapBruteForce, LdapBruteForceFields, LdapEventFields, LdapPlainText},
@@ -73,7 +73,6 @@ pub use self::{
     ssh::{BlockListSsh, BlockListSshFields},
     sysmon::WindowsThreat,
     tls::{BlockListTls, BlockListTlsFields, SuspiciousTlsTraffic},
-    tor::{HttpEventFields, TorConnection},
 };
 use super::{
     types::{Endpoint, HostNetworkGroup},
@@ -2719,7 +2718,7 @@ mod tests {
             method: "GET".to_string(),
             host: "example.com".to_string(),
             uri: "/uri/path".to_string(),
-            referer: "-".to_string(),
+            referrer: "-".to_string(),
             version: "1.1".to_string(),
             user_agent: "browser".to_string(),
             request_len: 100,
@@ -2749,7 +2748,7 @@ mod tests {
         let syslog_message = format!("{msg}");
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8""#
         );
 
         let dga = DomainGenerationAlgorithm::new(
@@ -2760,7 +2759,7 @@ mod tests {
         let dga_display = format!("{event}");
         assert_eq!(
             &dga_display,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8" triage_scores="""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8" triage_scores="""#
         );
     }
 
@@ -2858,7 +2857,7 @@ mod tests {
             method: "GET".to_string(),
             host: "example.com".to_string(),
             uri: "/uri/path".to_string(),
-            referer: "-".to_string(),
+            referrer: "-".to_string(),
             version: "1.1".to_string(),
             user_agent: "browser".to_string(),
             request_len: 100,
@@ -2893,7 +2892,7 @@ mod tests {
         let syslog_message = format!("{msg}");
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="HttpThreat" category="Reconnaissance" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" db_name="db" rule_id="12000" matched_to="match" cluster_id="1111" attack_kind="attack" confidence="0.8""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="HttpThreat" category="Reconnaissance" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" db_name="db" rule_id="12000" matched_to="match" cluster_id="1111" attack_kind="attack" confidence="0.8""#
         );
 
         let http_threat =
@@ -3403,7 +3402,7 @@ mod tests {
             message: "message".to_string(),
             renewal_time: 100,
             rebinding_time: 200,
-            class_id: vec![4, 5, 6],
+            class_id: "MSFT 5.0".to_string().into_bytes(),
             client_id_type: 1,
             client_id: vec![7, 8, 9],
             category: EventCategory::InitialAccess,
@@ -3422,7 +3421,7 @@ mod tests {
         let syslog_message = message.to_string();
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="04:05:06" client_id_type="1" client_id="07:08:09""#,
+            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="MSFT 5.0" client_id_type="1" client_id="07:08:09""#,
         );
 
         let block_list_dhcp = Event::BlockList(RecordType::Dhcp(BlockListDhcp::new(
@@ -3433,7 +3432,7 @@ mod tests {
 
         assert_eq!(
             &block_list_dhcp,
-            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="04:05:06" client_id_type="1" client_id="07:08:09" triage_scores="""#
+            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="MSFT 5.0" client_id_type="1" client_id="07:08:09" triage_scores="""#
         );
     }
 

diff --git a/src/event/bootp.rs b/src/event/bootp.rs
@@ -1,10 +1,39 @@
 use std::{fmt, net::IpAddr, num::NonZeroU8};
 
+use attrievent::attribute::{BootpAttr, RawEventKind};
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
 
-use super::{common::Match, EventCategory, TriagePolicy, TriageScore, MEDIUM};
-use crate::event::common::{to_hardware_address, triage_scores_to_string};
+use super::{common::Match, EventCategory, TriageScore, MEDIUM};
+use crate::event::common::{to_hardware_address, triage_scores_to_string, AttrValue};
+
+macro_rules! bootp_target_attr {
+    ($event: expr, $raw_event_attr: expr) => {{
+        if let RawEventKind::Bootp(attr) = $raw_event_attr {
+            let target_value = match attr {
+                BootpAttr::SrcAddr => AttrValue::Addr($event.src_addr),
+                BootpAttr::SrcPort => AttrValue::UInt($event.src_port.into()),
+                BootpAttr::DstAddr => AttrValue::Addr($event.dst_addr),
+                BootpAttr::DstPort => AttrValue::UInt($event.dst_port.into()),
+                BootpAttr::Proto => AttrValue::UInt($event.proto.into()),
+                BootpAttr::Op => AttrValue::UInt($event.op.into()),
+                BootpAttr::Htype => AttrValue::UInt($event.htype.into()),
+                BootpAttr::Hops => AttrValue::UInt($event.hops.into()),
+                BootpAttr::Xid => AttrValue::UInt($event.xid.into()),
+                BootpAttr::CiAddr => AttrValue::Addr($event.ciaddr),
+                BootpAttr::YiAddr => AttrValue::Addr($event.yiaddr),
+                BootpAttr::SiAddr => AttrValue::Addr($event.siaddr),
+                BootpAttr::GiAddr => AttrValue::Addr($event.giaddr),
+                BootpAttr::ChAddr => AttrValue::String(&$event.chaddr),
+                BootpAttr::SName => AttrValue::String(&$event.sname),
+                BootpAttr::File => AttrValue::String(&$event.file),
+            };
+            Some(target_value)
+        } else {
+            None
+        }
+    }};
+}
 
 #[derive(Serialize, Deserialize)]
 pub struct BlockListBootpFields {
@@ -73,7 +102,7 @@ pub struct BlockListBootp {
     pub yiaddr: IpAddr,
     pub siaddr: IpAddr,
     pub giaddr: IpAddr,
-    pub chaddr: Vec<u8>,
+    pub chaddr: String,
     pub sname: String,
     pub file: String,
     pub category: EventCategory,
@@ -99,7 +128,7 @@ impl fmt::Display for BlockListBootp {
             self.yiaddr.to_string(),
             self.siaddr.to_string(),
             self.giaddr.to_string(),
-            to_hardware_address(&self.chaddr),
+            self.chaddr,
             self.sname.to_string(),
             self.file.to_string(),
             triage_scores_to_string(self.triage_scores.as_ref())
@@ -126,7 +155,7 @@ impl BlockListBootp {
             yiaddr: fields.yiaddr,
             siaddr: fields.siaddr,
             giaddr: fields.giaddr,
-            chaddr: fields.chaddr,
+            chaddr: to_hardware_address(&fields.chaddr),
             sname: fields.sname,
             file: fields.file,
             category: fields.category,
@@ -176,7 +205,7 @@ impl Match for BlockListBootp {
         None
     }
 
-    fn score_by_packet_attr(&self, _triage: &TriagePolicy) -> f64 {
-        0.0
+    fn target_attribute(&self, raw_event_attr: RawEventKind) -> Option<AttrValue> {
+        bootp_target_attr!(self, raw_event_attr)
     }
 }