Add packet attribute triage function

- Add triage functionality for scoring with attributes of each protocol. - Add a new enum type `AttrValue`. - Add the `target_attribute` to the `Match` trait. - Implement `score_by_packet_attr` under `Match` trait. - Modify the `ValueKind` enum to support different types of input. - Remove the `tor` module file. The structures (`HttpEventFields`, `TorConnection`) and implementations within that module have been moved to `crate::event::http`. - Fix HTTP detection events to consistently use `referrer` instead of `referrer` and `referrer` interchangeably. - Change the type of fields in the detection event structure for some protocols. - `post_body`: `Vec<u8>` to `String`. - `chaddr`: `Vec<u8>` to `String`. - `class_id`: `Vec<u8>` to `String`. - `client_id`: `Vec<u8>` to `String`. Close: #354
petabi · Dec 4, 2024 · 9880f6d · 9880f6d
1 parent ac07e6d
commit 9880f6d
Show file tree

Hide file tree

Showing 27 changed files with 2,149 additions and 472 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,10 +7,34 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+
+- Added triage functionality for scoring with attributes of each protocol.
+  - Added a new enum type `AttrValue`. This type is used to convert the
+    attribute value of each protocol to its corresponding type to perform
+    comparison operations.
+  - Added the `target_attribute` to the `Match` trait to generate an `AttrValue`
+    from the field in all detection event.
+  - Implemented `score_by_packet_attr` under `Match` trait.
+
 ### Changed
 
 - Modified `Table::get_patterns` to take a reference to a slice of tuples
   instead of taking ownership of a vector, making the function more flexible.
+- Removed the `tor` module file. The structures (`HttpEventFields`,
+  `TorConnection`) and implementations within that module have been moved to
+  `crate::event::http`.
+- Fixed HTTP detection events to consistently use `referrer` instead of
+  `referrer` and `referrer` interchangeably.
+- Modified the `ValueKind` enum to support different types of input for packet
+  attribute triage.
+- Changed the type of fields in the detection event structure for some
+  protocols. This change allows users to see meaningful values directly without
+  having to do any special conversion for that field.
+  - `post_body`: `Vec<u8>` to `String`.
+  - `chaddr`: `Vec<u8>` to `String`.
+  - `class_id`: `Vec<u8>` to `String`.
+  - `client_id`: `Vec<u8>` to `String`.
 
 ## [0.32.0] - 2024-11-07
 
@@ -315,7 +339,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Changed the return type of `Store::network_map` to `IndexedTable<Network>`
   to enhance security by preventing direct exposure of internal structure.
 - Replaced `IndexedMap::get_by_id` function with `Indexed::get_by_id`, providing
-   a more structured and type-safe result.
+  a more structured and type-safe result.
   - Previously, the function returned a binary representation of the key-value
     pair: `Result<(Option<impl AsRef<[u8]>>, Option<impl AsRef<[u8]>>)>`.
   - Now, it returns `Result<Option<T>>`, where T is the entry type.
@@ -331,8 +355,8 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   more straightforward and human-readable format compared to the raw binary
   format exposed by `IndexSet`.
 - Replaced the `IndexedTable<Category>::get`, `IndexedTable<Qualifier>::get` and
- `IndexedTable<Status>::get` method with the more general function
- `IndexedTable<R>::get_by_id`. This change enhances flexibility by allowing
+  `IndexedTable<Status>::get` method with the more general function
+  `IndexedTable<R>::get_by_id`. This change enhances flexibility by allowing
   retrieval based on any type R rather than being limited to a specific category.
   Existing code using get for categories should be updated to use get_by_id with
   the appropriate type.
@@ -374,7 +398,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Added new functions to facilitate insert, remove, and update operations,
   ensuring a more controlled and secure template management.
 - Introduced `Structured`, `Unstructured`, `StructuredClusteringAlgorithm` and
- `UnstructuredClusteringAlgorithm` to describe data stored in `Table<Template>`.
+  `UnstructuredClusteringAlgorithm` to describe data stored in `Table<Template>`.
 - Introduced `TriageResponse` to describe data stored in `IndexedTable<TriageResponse>`.
 - Introduced `TriageResponseUpdate` to support `TriageResponse` record update.
 - Added new functions to facilitate insert, remove, and update operations,
@@ -406,7 +430,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   enhance security by preventing direct exposure of `Map`.
 - The `get_by_id` method in the `IndexedMap` struct has been updated to return a
   key-value pair (`(Vec<u8>, Vec<u8>)`) instead of just the value (`impl
-  AsRef<[u8]>`). This change accommodates scenarios where the information stored
+AsRef<[u8]>`). This change accommodates scenarios where the information stored
   in a key may not be present in the value for some Column Families. Previously,
   if you called `get_by_id` with a specific ID, you would receive the
   corresponding value as `Option<impl AsRef<[u8]>>`. Now, calling `get_by_id`
@@ -595,7 +619,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Removed `batch_info` and `scores` arguments from `Model::from_storage` function.
   These arguments were previously used for custom initialization of the
   `batch_info` and `scores` fields within the model. This change means that when
-  you create a model using `Model::from_storage`, the  `batch_info` and `scores`
+  you create a model using `Model::from_storage`, the `batch_info` and `scores`
   fields will now be initialized with their default values. If you previously
   relied on custom values for these fields, you will need to update your code accordingly.
 
@@ -683,6 +707,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   Please note that if you are currently using a version of the application
   earlier than 0.12.0, database migration support has been deprecated and will
   no longer be available.
+
   - Users on versions prior to 0.12.0 will need to manually manage their database
     schema updates if they choose to continue using these older versions.
   - We highly recommend upgrading to the latest version (0.12.0 or later) to
@@ -720,6 +745,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   code if you have been using the old naming convention. We apologise for any
   inconvenience this may cause, but we believe this change will bring greater
   consistency and readability to the codebase.
+
 - Removed `src_port` field from `FtpBruteForce` and `LdapBruteForce` events.
   to align with the event fields provided by hog.
 - Modified `LdapPlainText` fields to appropriate LDAP event fields from wrong

diff --git a/src/event.rs b/src/event.rs
@@ -20,7 +20,6 @@ mod smtp;
 mod ssh;
 mod sysmon;
 mod tls;
-mod tor;
 
 use std::{
     collections::HashMap,
@@ -57,8 +56,9 @@ pub use self::{
     },
     ftp::{BlockListFtp, FtpBruteForce, FtpBruteForceFields, FtpEventFields, FtpPlainText},
     http::{
-        BlockListHttp, BlockListHttpFields, DgaFields, DomainGenerationAlgorithm, HttpThreat,
-        HttpThreatFields, NonBrowser, RepeatedHttpSessions, RepeatedHttpSessionsFields,
+        BlockListHttp, BlockListHttpFields, DgaFields, DomainGenerationAlgorithm, HttpEventFields,
+        HttpThreat, HttpThreatFields, NonBrowser, RepeatedHttpSessions, RepeatedHttpSessionsFields,
+        TorConnection,
     },
     kerberos::{BlockListKerberos, BlockListKerberosFields},
     ldap::{BlockListLdap, LdapBruteForce, LdapBruteForceFields, LdapEventFields, LdapPlainText},
@@ -73,7 +73,6 @@ pub use self::{
     ssh::{BlockListSsh, BlockListSshFields},
     sysmon::WindowsThreat,
     tls::{BlockListTls, BlockListTlsFields, SuspiciousTlsTraffic},
-    tor::{HttpEventFields, TorConnection},
 };
 use super::{
     types::{Endpoint, HostNetworkGroup},
@@ -2719,7 +2718,7 @@ mod tests {
             method: "GET".to_string(),
             host: "example.com".to_string(),
             uri: "/uri/path".to_string(),
-            referer: "-".to_string(),
+            referrer: "-".to_string(),
             version: "1.1".to_string(),
             user_agent: "browser".to_string(),
             request_len: 100,
@@ -2749,7 +2748,7 @@ mod tests {
         let syslog_message = format!("{msg}");
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" source="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" source="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8""#
         );
 
         let dga = DomainGenerationAlgorithm::new(
@@ -2760,7 +2759,7 @@ mod tests {
         let dga_display = format!("{event}");
         assert_eq!(
             &dga_display,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" source="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8" triage_scores="""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" source="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8" triage_scores="""#
         );
     }
 
@@ -2858,7 +2857,7 @@ mod tests {
             method: "GET".to_string(),
             host: "example.com".to_string(),
             uri: "/uri/path".to_string(),
-            referer: "-".to_string(),
+            referrer: "-".to_string(),
             version: "1.1".to_string(),
             user_agent: "browser".to_string(),
             request_len: 100,
@@ -2893,7 +2892,7 @@ mod tests {
         let syslog_message = format!("{msg}");
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="HttpThreat" category="Reconnaissance" source="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" db_name="db" rule_id="12000" matched_to="match" cluster_id="1111" attack_kind="attack" confidence="0.8""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="HttpThreat" category="Reconnaissance" source="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" db_name="db" rule_id="12000" matched_to="match" cluster_id="1111" attack_kind="attack" confidence="0.8""#
         );
 
         let http_threat =
@@ -3403,7 +3402,7 @@ mod tests {
             message: "message".to_string(),
             renewal_time: 100,
             rebinding_time: 200,
-            class_id: vec![4, 5, 6],
+            class_id: "MSFT 5.0".to_string().into_bytes(),
             client_id_type: 1,
             client_id: vec![7, 8, 9],
             category: EventCategory::InitialAccess,
@@ -3422,7 +3421,7 @@ mod tests {
         let syslog_message = message.to_string();
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" source="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="04:05:06" client_id_type="1" client_id="07:08:09""#,
+            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" source="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="MSFT 5.0" client_id_type="1" client_id="07:08:09""#,
         );
 
         let block_list_dhcp = Event::BlockList(RecordType::Dhcp(BlockListDhcp::new(
@@ -3433,7 +3432,7 @@ mod tests {
 
         assert_eq!(
             &block_list_dhcp,
-            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" source="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="04:05:06" client_id_type="1" client_id="07:08:09" triage_scores="""#
+            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" source="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="MSFT 5.0" client_id_type="1" client_id="07:08:09" triage_scores="""#
         );
     }
 

diff --git a/src/event/bootp.rs b/src/event/bootp.rs
@@ -2,9 +2,71 @@ use std::{fmt, net::IpAddr, num::NonZeroU8};
 
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
+use strum_macros::EnumString;
+
+use super::{common::Match, EventCategory, TriageScore, MEDIUM};
+use crate::event::common::{to_hardware_address, triage_scores_to_string, AttrValue};
+
+macro_rules! bootp_target_attr {
+    ($event: expr, $proto_attr: expr) => {{
+        let target_value = match $proto_attr {
+            BootpAttr::SrcAddr => AttrValue::Addr($event.src_addr),
+            BootpAttr::SrcPort => AttrValue::UInt($event.src_port.into()),
+            BootpAttr::DstAddr => AttrValue::Addr($event.dst_addr),
+            BootpAttr::DstPort => AttrValue::UInt($event.dst_port.into()),
+            BootpAttr::Proto => AttrValue::UInt($event.proto.into()),
+            BootpAttr::Op => AttrValue::UInt($event.op.into()),
+            BootpAttr::Htype => AttrValue::UInt($event.htype.into()),
+            BootpAttr::Hops => AttrValue::UInt($event.hops.into()),
+            BootpAttr::Xid => AttrValue::UInt($event.xid.into()),
+            BootpAttr::CiAddr => AttrValue::Addr($event.ciaddr),
+            BootpAttr::YiAddr => AttrValue::Addr($event.yiaddr),
+            BootpAttr::SiAddr => AttrValue::Addr($event.siaddr),
+            BootpAttr::GiAddr => AttrValue::Addr($event.giaddr),
+            BootpAttr::ChAddr => AttrValue::String(&$event.chaddr),
+            BootpAttr::SName => AttrValue::String(&$event.sname),
+            BootpAttr::File => AttrValue::String(&$event.file),
+        };
+        Some(target_value)
+    }};
+}
 
-use super::{common::Match, EventCategory, TriagePolicy, TriageScore, MEDIUM};
-use crate::event::common::{to_hardware_address, triage_scores_to_string};
+#[allow(clippy::module_name_repetitions)]
+#[derive(Debug, EnumString, PartialEq)]
+pub enum BootpAttr {
+    #[strum(serialize = "bootp-id.orig_h")]
+    SrcAddr,
+    #[strum(serialize = "bootp-id.orig_p")]
+    SrcPort,
+    #[strum(serialize = "bootp-id.resp_h")]
+    DstAddr,
+    #[strum(serialize = "bootp-id.resp_p")]
+    DstPort,
+    #[strum(serialize = "bootp-proto")]
+    Proto,
+    #[strum(serialize = "bootp-op")]
+    Op,
+    #[strum(serialize = "bootp-htype")]
+    Htype,
+    #[strum(serialize = "bootp-hops")]
+    Hops,
+    #[strum(serialize = "bootp-xid")]
+    Xid,
+    #[strum(serialize = "bootp-ciaddr")]
+    CiAddr,
+    #[strum(serialize = "bootp-yiaddr")]
+    YiAddr,
+    #[strum(serialize = "bootp-siaddr")]
+    SiAddr,
+    #[strum(serialize = "bootp-giaddr")]
+    GiAddr,
+    #[strum(serialize = "bootp-chaddr")]
+    ChAddr,
+    #[strum(serialize = "bootp-sname")]
+    SName,
+    #[strum(serialize = "bootp-file")]
+    File,
+}
 
 #[derive(Serialize, Deserialize)]
 pub struct BlockListBootpFields {
@@ -73,7 +135,7 @@ pub struct BlockListBootp {
     pub yiaddr: IpAddr,
     pub siaddr: IpAddr,
     pub giaddr: IpAddr,
-    pub chaddr: Vec<u8>,
+    pub chaddr: String,
     pub sname: String,
     pub file: String,
     pub category: EventCategory,
@@ -99,7 +161,7 @@ impl fmt::Display for BlockListBootp {
             self.yiaddr.to_string(),
             self.siaddr.to_string(),
             self.giaddr.to_string(),
-            to_hardware_address(&self.chaddr),
+            self.chaddr,
             self.sname.to_string(),
             self.file.to_string(),
             triage_scores_to_string(&self.triage_scores)
@@ -126,7 +188,7 @@ impl BlockListBootp {
             yiaddr: fields.yiaddr,
             siaddr: fields.siaddr,
             giaddr: fields.giaddr,
-            chaddr: fields.chaddr,
+            chaddr: to_hardware_address(&fields.chaddr),
             sname: fields.sname,
             file: fields.file,
             category: fields.category,
@@ -135,7 +197,7 @@ impl BlockListBootp {
     }
 }
 
-impl Match for BlockListBootp {
+impl Match<BootpAttr> for BlockListBootp {
     fn src_addr(&self) -> IpAddr {
         self.src_addr
     }
@@ -176,7 +238,7 @@ impl Match for BlockListBootp {
         None
     }
 
-    fn score_by_packet_attr(&self, _triage: &TriagePolicy) -> f64 {
-        0.0
+    fn target_attribute(&self, proto_attr: BootpAttr) -> Option<AttrValue> {
+        bootp_target_attr!(self, proto_attr)
     }
 }