Add packet attribute triage function

- Add triage functionality for scoring with attributes of each raw event. - Introduced a new crate attrievent. The kinds of raw events and their attributes change as our software evolves. The purpose of attrievent is to provide a comprehensive list of attributes for both review and the UI simultaneously. - Add the `to_attr_value` to the `Match` trait. - Implement `score_by_attr` under `Match` trait. - Modify the `ValueKind` enum to support different types of input. - Remove the `tor` module file. The structures (`HttpEventFields`, `TorConnection`) and implementations within that module have been moved to `crate::event::http`. - Fix HTTP detection events to consistently use `referrer` instead of `referrer` and `referer` interchangeably. - Change the type of fields in the detection event structure for some raw event. - `post_body`: `Vec<u8>` to `String`. - `chaddr`: `Vec<u8>` to `String`. - `class_id`: `Vec<u8>` to `String`. - `client_id`: `Vec<u8>` to `String`. Close: #354
petabi · Dec 20, 2024 · b1eaa7e · b1eaa7e
1 parent 61daf7b
commit b1eaa7e
Show file tree

Hide file tree

Showing 28 changed files with 1,509 additions and 437 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,39 @@ file is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and
 this project adheres to [Semantic
 Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- Added triage functionality for scoring with attributes of each raw event.
+  - Introduced a new crate attrievent. The kinds of raw events and their
+    attributes change as our software evolves. The purpose of attrievent is to
+    provide a comprehensive list of attributes for both review and the UI
+    simultaneously.
+  - Added a new enum type `AttrValue`. This type is used to convert the
+    attribute value of each raw event to its corresponding type to perform
+    comparison operations.
+  - Added the `to_attr_value` to the `Match` trait to generate an `AttrValue`
+    from the field in all detection event.
+  - Implemented `score_by_attr` under `Match` trait.
+
+### Changed
+
+- Removed the `tor` module file. The structures (`HttpEventFields`,
+  `TorConnection`) and implementations within that module have been moved to
+  `crate::event::http`.
+- Fixed HTTP detection events to consistently use `referrer` instead of
+  `referrer` and `referer` interchangeably.
+- Modified the `ValueKind` enum to support different types of input for packet
+  attribute triage.
+- Changed the type of fields in the detection event structure for some raw
+  event. This change allows users to see meaningful values directly without
+  having to do any special conversion for that field.
+  - `post_body`(HTTP): `Vec<u8>` to `String`.
+  - `chaddr`(BOOTP): `Vec<u8>` to `String`.
+  - `class_id`(DHCP): `Vec<u8>` to `String`.
+  - `client_id`(DHCP): `Vec<u8>` to `String`.
+
 ## [0.33.0] - 2024-12-12
 
 ### Changed
@@ -327,7 +360,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Changed the return type of `Store::network_map` to `IndexedTable<Network>`
   to enhance security by preventing direct exposure of internal structure.
 - Replaced `IndexedMap::get_by_id` function with `Indexed::get_by_id`, providing
-   a more structured and type-safe result.
+  a more structured and type-safe result.
   - Previously, the function returned a binary representation of the key-value
     pair: `Result<(Option<impl AsRef<[u8]>>, Option<impl AsRef<[u8]>>)>`.
   - Now, it returns `Result<Option<T>>`, where T is the entry type.
@@ -343,8 +376,8 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   more straightforward and human-readable format compared to the raw binary
   format exposed by `IndexSet`.
 - Replaced the `IndexedTable<Category>::get`, `IndexedTable<Qualifier>::get` and
- `IndexedTable<Status>::get` method with the more general function
- `IndexedTable<R>::get_by_id`. This change enhances flexibility by allowing
+  `IndexedTable<Status>::get` method with the more general function
+  `IndexedTable<R>::get_by_id`. This change enhances flexibility by allowing
   retrieval based on any type R rather than being limited to a specific category.
   Existing code using get for categories should be updated to use get_by_id with
   the appropriate type.
@@ -386,7 +419,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Added new functions to facilitate insert, remove, and update operations,
   ensuring a more controlled and secure template management.
 - Introduced `Structured`, `Unstructured`, `StructuredClusteringAlgorithm` and
- `UnstructuredClusteringAlgorithm` to describe data stored in `Table<Template>`.
+  `UnstructuredClusteringAlgorithm` to describe data stored in `Table<Template>`.
 - Introduced `TriageResponse` to describe data stored in `IndexedTable<TriageResponse>`.
 - Introduced `TriageResponseUpdate` to support `TriageResponse` record update.
 - Added new functions to facilitate insert, remove, and update operations,
@@ -418,7 +451,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   enhance security by preventing direct exposure of `Map`.
 - The `get_by_id` method in the `IndexedMap` struct has been updated to return a
   key-value pair (`(Vec<u8>, Vec<u8>)`) instead of just the value (`impl
-  AsRef<[u8]>`). This change accommodates scenarios where the information stored
+AsRef<[u8]>`). This change accommodates scenarios where the information stored
   in a key may not be present in the value for some Column Families. Previously,
   if you called `get_by_id` with a specific ID, you would receive the
   corresponding value as `Option<impl AsRef<[u8]>>`. Now, calling `get_by_id`
@@ -607,7 +640,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Removed `batch_info` and `scores` arguments from `Model::from_storage` function.
   These arguments were previously used for custom initialization of the
   `batch_info` and `scores` fields within the model. This change means that when
-  you create a model using `Model::from_storage`, the  `batch_info` and `scores`
+  you create a model using `Model::from_storage`, the `batch_info` and `scores`
   fields will now be initialized with their default values. If you previously
   relied on custom values for these fields, you will need to update your code accordingly.
 
@@ -695,6 +728,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   Please note that if you are currently using a version of the application
   earlier than 0.12.0, database migration support has been deprecated and will
   no longer be available.
+
   - Users on versions prior to 0.12.0 will need to manually manage their database
     schema updates if they choose to continue using these older versions.
   - We highly recommend upgrading to the latest version (0.12.0 or later) to
@@ -732,6 +766,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   code if you have been using the old naming convention. We apologise for any
   inconvenience this may cause, but we believe this change will bring greater
   consistency and readability to the codebase.
+
 - Removed `src_port` field from `FtpBruteForce` and `LdapBruteForce` events.
   to align with the event fields provided by hog.
 - Modified `LdapPlainText` fields to appropriate LDAP event fields from wrong
@@ -740,6 +775,7 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Modified `FtpBruteForce` by adding an `is_internal` field which is a boolean
   indicating whether it is internal or not.
 
+[Unreleased]: https://github.com/aicers/review-database/compare/0.33.0...main
 [0.33.0]: https://github.com/petabi/review-database/compare/0.32.0...0.33.0
 [0.32.0]: https://github.com/petabi/review-database/compare/0.31.0...0.32.0
 [0.31.0]: https://github.com/petabi/review-database/compare/0.30.0...0.31.0

diff --git a/Cargo.toml b/Cargo.toml
@@ -52,6 +52,7 @@ tokio = { version = "1", features = ["macros"] }
 tokio-postgres-rustls = "0.13"
 toml = "0.8.13"
 tracing = "0.1"
+attrievent = { git = "https://github.com/aicers/attrievent.git", rev = "2982a7b" }
 
 [dev-dependencies]
 tempfile = "3"

diff --git a/src/event.rs b/src/event.rs
@@ -20,7 +20,6 @@ mod smtp;
 mod ssh;
 mod sysmon;
 mod tls;
-mod tor;
 
 use std::{
     collections::HashMap,
@@ -57,8 +56,9 @@ pub use self::{
     },
     ftp::{BlockListFtp, FtpBruteForce, FtpBruteForceFields, FtpEventFields, FtpPlainText},
     http::{
-        BlockListHttp, BlockListHttpFields, DgaFields, DomainGenerationAlgorithm, HttpThreat,
-        HttpThreatFields, NonBrowser, RepeatedHttpSessions, RepeatedHttpSessionsFields,
+        BlockListHttp, BlockListHttpFields, DgaFields, DomainGenerationAlgorithm, HttpEventFields,
+        HttpThreat, HttpThreatFields, NonBrowser, RepeatedHttpSessions, RepeatedHttpSessionsFields,
+        TorConnection,
     },
     kerberos::{BlockListKerberos, BlockListKerberosFields},
     ldap::{BlockListLdap, LdapBruteForce, LdapBruteForceFields, LdapEventFields, LdapPlainText},
@@ -73,7 +73,6 @@ pub use self::{
     ssh::{BlockListSsh, BlockListSshFields},
     sysmon::WindowsThreat,
     tls::{BlockListTls, BlockListTlsFields, SuspiciousTlsTraffic},
-    tor::{HttpEventFields, TorConnection},
 };
 use super::{
     types::{Endpoint, HostNetworkGroup},
@@ -2719,7 +2718,7 @@ mod tests {
             method: "GET".to_string(),
             host: "example.com".to_string(),
             uri: "/uri/path".to_string(),
-            referer: "-".to_string(),
+            referrer: "-".to_string(),
             version: "1.1".to_string(),
             user_agent: "browser".to_string(),
             request_len: 100,
@@ -2749,7 +2748,7 @@ mod tests {
         let syslog_message = format!("{msg}");
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8""#
         );
 
         let dga = DomainGenerationAlgorithm::new(
@@ -2760,7 +2759,7 @@ mod tests {
         let dga_display = format!("{event}");
         assert_eq!(
             &dga_display,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8" triage_scores="""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="DomainGenerationAlgorithm" category="CommandAndControl" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" confidence="0.8" triage_scores="""#
         );
     }
 
@@ -2858,7 +2857,7 @@ mod tests {
             method: "GET".to_string(),
             host: "example.com".to_string(),
             uri: "/uri/path".to_string(),
-            referer: "-".to_string(),
+            referrer: "-".to_string(),
             version: "1.1".to_string(),
             user_agent: "browser".to_string(),
             request_len: 100,
@@ -2893,7 +2892,7 @@ mod tests {
         let syslog_message = format!("{msg}");
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T00:01:01+00:00" event_kind="HttpThreat" category="Reconnaissance" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" db_name="db" rule_id="12000" matched_to="match" cluster_id="1111" attack_kind="attack" confidence="0.8""#
+            r#"time="1970-01-01T00:01:01+00:00" event_kind="HttpThreat" category="Reconnaissance" sensor="collector1" src_addr="127.0.0.1" src_port="10000" dst_addr="127.0.0.2" dst_port="80" proto="6" duration="1000" method="GET" host="example.com" uri="/uri/path" referrer="-" version="1.1" user_agent="browser" request_len="100" response_len="100" status_code="200" status_msg="-" username="-" password="-" cookie="cookie" content_encoding="encoding type" content_type="content type" cache_control="no cache" orig_filenames="a1,a2" orig_mime_types="" resp_filenames="" resp_mime_types="b1,b2" post_body="1234567890..." state="" db_name="db" rule_id="12000" matched_to="match" cluster_id="1111" attack_kind="attack" confidence="0.8""#
         );
 
         let http_threat =
@@ -3403,7 +3402,7 @@ mod tests {
             message: "message".to_string(),
             renewal_time: 100,
             rebinding_time: 200,
-            class_id: vec![4, 5, 6],
+            class_id: "MSFT 5.0".to_string().into_bytes(),
             client_id_type: 1,
             client_id: vec![7, 8, 9],
             category: EventCategory::InitialAccess,
@@ -3422,7 +3421,7 @@ mod tests {
         let syslog_message = message.to_string();
         assert_eq!(
             &syslog_message,
-            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="04:05:06" client_id_type="1" client_id="07:08:09""#,
+            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="MSFT 5.0" client_id_type="1" client_id="07:08:09""#,
         );
 
         let block_list_dhcp = Event::BlockList(RecordType::Dhcp(BlockListDhcp::new(
@@ -3433,7 +3432,7 @@ mod tests {
 
         assert_eq!(
             &block_list_dhcp,
-            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="04:05:06" client_id_type="1" client_id="07:08:09" triage_scores="""#
+            r#"time="1970-01-01T01:01:01+00:00" event_kind="BlockListDhcp" category="InitialAccess" sensor="collector1" src_addr="127.0.0.1" src_port="68" dst_addr="127.0.0.2" dst_port="67" proto="17" last_time="100" msg_type="1" ciaddr="127.0.0.5" yiaddr="127.0.0.6" siaddr="127.0.0.7" giaddr="127.0.0.8" subnet_mask="255.255.255.0" router="127.0.0.1" domain_name_server="127.0.0.1" req_ip_addr="127.0.0.100" lease_time="100" server_id="127.0.0.1" param_req_list="1,2,3" message="message" renewal_time="100" rebinding_time="200" class_id="MSFT 5.0" client_id_type="1" client_id="07:08:09" triage_scores="""#
         );
     }
 

diff --git a/src/event/bootp.rs b/src/event/bootp.rs
@@ -1,10 +1,39 @@
 use std::{fmt, net::IpAddr, num::NonZeroU8};
 
+use attrievent::attribute::{BootpAttr, RawEventAttrKind};
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
 
-use super::{common::Match, EventCategory, TriagePolicy, TriageScore, MEDIUM};
-use crate::event::common::{to_hardware_address, triage_scores_to_string};
+use super::{common::Match, EventCategory, TriageScore, MEDIUM};
+use crate::event::common::{to_hardware_address, triage_scores_to_string, AttrValue};
+
+macro_rules! bootp_target_attr {
+    ($event: expr, $raw_event_attr: expr) => {{
+        if let RawEventAttrKind::Bootp(attr) = $raw_event_attr {
+            let target_value = match attr {
+                BootpAttr::SrcAddr => AttrValue::Addr($event.src_addr),
+                BootpAttr::SrcPort => AttrValue::UInt($event.src_port.into()),
+                BootpAttr::DstAddr => AttrValue::Addr($event.dst_addr),
+                BootpAttr::DstPort => AttrValue::UInt($event.dst_port.into()),
+                BootpAttr::Proto => AttrValue::UInt($event.proto.into()),
+                BootpAttr::Op => AttrValue::UInt($event.op.into()),
+                BootpAttr::Htype => AttrValue::UInt($event.htype.into()),
+                BootpAttr::Hops => AttrValue::UInt($event.hops.into()),
+                BootpAttr::Xid => AttrValue::UInt($event.xid.into()),
+                BootpAttr::CiAddr => AttrValue::Addr($event.ciaddr),
+                BootpAttr::YiAddr => AttrValue::Addr($event.yiaddr),
+                BootpAttr::SiAddr => AttrValue::Addr($event.siaddr),
+                BootpAttr::GiAddr => AttrValue::Addr($event.giaddr),
+                BootpAttr::ChAddr => AttrValue::String(&$event.chaddr),
+                BootpAttr::SName => AttrValue::String(&$event.sname),
+                BootpAttr::File => AttrValue::String(&$event.file),
+            };
+            Some(target_value)
+        } else {
+            None
+        }
+    }};
+}
 
 #[derive(Serialize, Deserialize)]
 pub struct BlockListBootpFields {
@@ -73,7 +102,7 @@ pub struct BlockListBootp {
     pub yiaddr: IpAddr,
     pub siaddr: IpAddr,
     pub giaddr: IpAddr,
-    pub chaddr: Vec<u8>,
+    pub chaddr: String,
     pub sname: String,
     pub file: String,
     pub category: EventCategory,
@@ -99,7 +128,7 @@ impl fmt::Display for BlockListBootp {
             self.yiaddr.to_string(),
             self.siaddr.to_string(),
             self.giaddr.to_string(),
-            to_hardware_address(&self.chaddr),
+            self.chaddr,
             self.sname.to_string(),
             self.file.to_string(),
             triage_scores_to_string(self.triage_scores.as_ref())
@@ -126,7 +155,7 @@ impl BlockListBootp {
             yiaddr: fields.yiaddr,
             siaddr: fields.siaddr,
             giaddr: fields.giaddr,
-            chaddr: fields.chaddr,
+            chaddr: to_hardware_address(&fields.chaddr),
             sname: fields.sname,
             file: fields.file,
             category: fields.category,
@@ -176,7 +205,7 @@ impl Match for BlockListBootp {
         None
     }
 
-    fn score_by_packet_attr(&self, _triage: &TriagePolicy) -> f64 {
-        0.0
+    fn to_attr_value(&self, raw_event_attr: RawEventAttrKind) -> Option<AttrValue> {
+        bootp_target_attr!(self, raw_event_attr)
     }
 }