2.4.60 #390
Merged
2.4.60 #390
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contains hardcoded config values in infohandler, mock data inside the JS. Very much a prototype.
Walked back playbooks to focus on detections. Made detections more data driven. Added form validation when creating a detection. TODO: form validation when editing. Added Duplicate functionality to the API/UI, added Bulk enable/disable to API. Touched the surface on how detections will sync.
Enabling/Disabling rules that use flowbits is different than rules that don't. Parsing needs testing, currently failing because the rule I'm testing with uses angled quotes instead of straight quotes. Can now delete a Detection from the UI. UI now performs some light validation before saving a Detection.
Properly adds SIDs to disabled file. MinLength now set in UI to match server requirement of 5. After creating a detection, you're taken to the edit page for that detection instead of being left at a filled in create page. If creating fails, a banner is shown. Case insensitive sidExtractor. The extractSID function now returns nil if more than 1 SID is specified. Tests for pure functions.
Added MemConfigStore to facilitate testing. Test rule parsing, enabling/disabling of rules, and various helper functions.
Moved Casestore functions relating to Detections to a brand new Detectionstore. TODO: TestSyncSuricata needs more tests, it currently only covers the bare minimum. Also, more testing needs to take place around what happens when the detections module is disabled.
Converted repeated strings to fixtures. Fleshed out the rest of SyncSuricata's tests. Added validation tests.
Removed permission checks left over from creating ElasticDetectionstore from ElasticCasestore. Jerry rigged the config to always have a `suricataengine` module section so I don't have to fight salt. Expanded on detection severity types. Expanded the DetectionEngine interface to support how we're importing community rules. First pass at a SyncCommunityRules. Solved some issues around ParseSuricataRule and whitespace.
Removed the sync endpoint from detections handler. Instead of so-rule-update attempting to upload the file or trigger a sync, the SuricataEngine module will watch files on disks for changes and respond to them. The DetectionEngine interface was cleaned up in response to this change. All sync logic was moved the SuricataEngine module. The module now maintains a long-lived goroutine that checks a configurable file at a configurable interval for changes and applies them when seen. A fingerprint of the rules file is saved in a configurable location. TODO: Tests.
The context that the server initializes is being put to use and slightly modified. As I find new permissions I need, I'm adding roles to cover them. When the dust settles, I'll re-evaluate. Search criteria now determine what permissions they need to check the user account for. Hints can be given but the search criteria's index and kind are tested to see if alternate permissions need to be checked. detection/read and detection/write were quickly added to the rbac/permissions file. The roles they're attached to has not been finalized and will almost certainly change. Lengthened the allowed title as many community rules have titles longer than 100 chars. When RoundTrip'ing with ElasticTransport, do not add the es-security-runas-user header when the server's agent is in the ctx. Added 2 new test rules to TestValidate inspired by community rules that couldn't parse but should've. Successfully imported 30,000+ community rules!
Uses elasticsearch's `search_after` to page through results when the max documents of -1 is passed in.
Added logic to disallow editing of community rules. Adjusted detection's form validation rules so that community rules aren't validated at the form level. Removed "Details" Detection tab and reordered the controls under it to the "Summary" and "Signature" tabs. Rearranged some tabs. Cleaned up the strings used in dropdowns (engine, severity) with capitalization. Cleaner UI, but the correct casing is still passed over the wire. Refactored PublicID's "Generate" button to be "Extract" so that it fills the field using the SID in the rule. May eventually make this field readonly except for this modification. Updated infohandler's response to use the same severity values the rules use. Converted more strings to i18n references. `indexDocument` and `deleteDocument` on the ElasticEventstore now pass the ctx in. This removes the WARN log message about making a request without a user. Needs more thorough testing to be sure there's no unwanted side effects.
New library: go-git. We use it to clone/pull the sigma repo. Prevent community rules from being updated through the detection CRUD endpoints. Improved change detection in suricata's community importer. If a community rule's content hasn't changed then it isn't updated, significantly improving the time it takes to update community rules. Improved support for sort criteria when querying elasticsearch. Linting.
ElastAlert community rules are imported from zips in SigmaHQ's release page on github. The configured packages are downloaded, parsed, and fingerprinted. Currently the sigma converter isn't completely implemented until some networking issues are fixed. GetAllCommunitySIDs can now filter by engine (or no filter if nil is passed). Updated detection severities to match our pickiest engine so far: elastalert. Added ID field to SigmaRules. Although it's not required, all community rules have one. This is the corresponding field to a detection's PublicID. TODO: tests
Backend: Removed go-github. Instead, we're using a configurable url template. Updated config values to impact how the UI renders. Replaced all references of github.com/tj/assert with references to github.com/stretchr/testify/assert for consistency. Fixed an issue where errors resulted in more errors when syncing local detections in elastalert. ElastAlert's SyncLocalDetections now implemented. ElastAlert's syncCommunityDetections is further implemented. Fixed a bug where syncing community detections resulted in purging local detections. Tests. Refactor to make some functions more testable. UI: Hide charts when on detections page in advanced mode. Removed auto-grow from rule text area. For large rules, modifying the rule would adjust the text area size which would adjust the browser's scroll position sometimes with counter productive effects. Include js-yaml library to parse yaml (i.e. sigma) to retrieve values. Used to extract the PublicID field from a sigma rule. More uses to come.
`make([]any, x, x)` only needs to specify len, cap will match. `x == nil || len(x) == 0` only the len check is necessary. len(nil) is 0. `time.Now().Sub(...)` should be `time.Since(...)`. Define, init, return simplified to return. Unlike other languages, go doesn't require `break` statements at the end of cases in a switch.
SeverityTranslations. A map used to translate one engine's severity levels to the levels we support in SOC's detections. A lot of UI work around CRUDing overrides in the Tuning tab of a detection. A little cleanup in hunt.js. Removed helper functions from the early days of detections. Could/should have been removed a long time ago. Initial logic for applying overrides when syncing detections started, but the modified threshold file isn't updated yet until I revisit and test everything.
Overrides can now be enabled/disabled. When saving a detection, add the saving overlay so multiple edits can't take place at once. Apply CustomFilter overrides when wrapping. Save threshold modifications to disk after syncing local detections and their overrides in suricata.
Expanding one override would expand all the rows. Fixed. Expanded CIDR checking to include IPv6. Changed OverrideParameters yaml tags: genId => gen_id, thresholdType => type, and no yaml output for CustomFilter.
Track is a field for both Suppress and Threshold type overrides but it contains different enum values depending on which of those two types, basically Threshold doesn't have `by_either`. When saving a detection, update the UpdatedAt field for an override when the old detection doesn't have an override with the exact same values regardless of order.
ElastAlert now enables first seen detections from the community. The infohandler now returns more predetermined search queries.
Also updated new chi references to chi/v5
Refactored the ElastAlertEngine to use a new interface for interacting with external resources (Disk and Network). This allows for mocking these resources in the new tests. Also updated Suricata's tests to include the new thresholding config value used by overrides.
To thoroughly test the parser, the process loads all the stored rules from the yara rules folder but does not attempt to sync them to elasticsearch. Also includes tests with hand crafted queries with "features" found in the community rules. Currently the only error this should run across is a legitimate error in a source rule: https://github.com/Security-Onion-Solutions/securityonion-yara/blob/master/yara/cn_pentestset_webshells.yar#L744
Detections can now be selected for bulk actions. The Select All checkbox toggles between selecting the entire current page and clearing all selection. If the current page is selected, a prompt to select ALL of the results becomes available. Currently the only actions that can be taken are the enabling or disabling of selected detections.
If picking individual IDs, then those IDs are sent up to the API to be modified. If ALL are selected, then the query is sent and detections matching the query will be modified.
A few component tests were added to hunt.test.js testing the SelectAll logic. Cypress tests will be added later.
UpdateDetectionField has been updated to update a detection field (i.e. without having the entire object) faster. Also should no longer be limited to one field although the functionality is not tested or implemented.
The ElasticDetectionstore now gets an ElasticSearch client passed into it so actions don't have to be run against Eventstore.
An error was fixed in the suricata parser. Better care is taken around modifying settings files that may be empty.
Settings' validator that used to check for Jinja now checks for opening AND closing jinja tags to prevent accidental false positives such as the suricata rule that contained `{%` in a URL.
When hunting, the daterange for detections will always be from the unix epoch to now. This helps us treat detections like a traditional model instead of a timescale DB entry.
Submitting a bulk operation against detections now results in a 202 (Accepted) and the work is done in a goroutine. At the end, the results are broadcast back to the client.
…th SigmaCLI There's some side effects because the default `yaraRulesFolder` is still being used to store non-compiled rules by salt. Updated tests around the new ExecCommand added to IOManagers. This also includes ElastAlert switching to using the sigma cli instead of the sigconverter.io container. All references to the container have been removed.
Fixed a bug where 2 slashes before ending a string resulted in the parser not recognizing that the string terminated. Updated "Parentheses in Unquoted Option" test to also test this case.
pcap improvements
This is necessary for other unit tests and general stability.
…-config-fix Allow For Zero Strelka Rules Repos
improve reboot indicators
allow sensoroni modules to succeed if at least one provided data
Don't show the title on the Detection Create page. Added a few classes and IDs to help tests. When extracting references, check URLs with Regex so we don't turn plain text into a hyperlink. Fixed a bug where extracting logic wouldn't extract anything if the unindent process didn't unindent anything. When extracting details from sigma rules, extract description too if present. Better logic around extracting yara logic. The `strings:` section is useful but not required. If not found, fallback to looking for `condition:` which is required.
Stopped prefilling the author field when we create the detection model. Instead, add the author to new detections right before we save them. Fixed an issue where non-URL Suricata references where showing up as hyperlinks. Some whitespace trimming in a few places. Fix for extractedLogic not being set if unindenting is unnecessary. The language dropdown entries are now displayed capitalized. Values are still lowercase. Cleaning up overrides turned into a function for testing.
…nstore. The handler should check for all the ways the request will fail. When everything succeeds, the store should do all the manipulation. This also prevents a redundant call to the DB for the existing detection. Removed "tautologies" that the linter was pointing out. Moved another validator from Casestore to Detectionstore for validating a Detection's tags. Fixed a bug when validating the ruleset of a detection. When validating a detection, check the language against the engine to be sure it's a valid pairing. DoesTemplateExist now considers any 2XX successful instead of just 200. Removed a couple redundant auth checks. Updated Casestore tests to have unique test names with Detectionstore and to fix tests that fail from an introduced change. ValidateStringArray was assuming the label was "Tags" even though a label was passed in, but the value was "tags". Detectionstore tests. TODO: More Detectionstore tests.
…on-tests Cogburn/detection tests
Also updated the name of jsyaml's code file to include the version number of the package.
…import-fix Fix for jsyaml reference in tests
New default values align with the values after a fresh SO installation. Also includes a SuricataEngine update so AllSettings are only retrieved once per sync cycle. This removes 1 redundant call.
…ons-defaults Better Defaults in Detection Engines
…links Add NIDS Tuning Links back in
check bidirectional pcap
add more logging
Improve Suri PCAP parsing performance
Added the version number to jsyaml a few commits ago. But that was incomplete: 1) I changed the filename and added the import in test_common, but I forgot to update the script tag in index.html. 2) I should have made the same change to the external lz string library. jsyaml now has the version number in the filename everywhere it's referenced. The LZString library has been given the same treatment. The 1.5.0 was added to the filename, the script tag reference was updated, and the LZString library has been added to global in test_common in line with the others.
Add Version Number to External JS Filenames
dougburks
approved these changes
Mar 20, 2024
m0duspwnens
approved these changes
Mar 20, 2024
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.