forked from cisagov/Malcolm
-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
improve the extracted_files download page #329
Labels
carving
Relating to carving (extraction) of files from traffic and the scanning of those files
enhancement
New feature or request
Milestone
Comments
mmguero
added
carving
Relating to carving (extraction) of files from traffic and the scanning of those files
enhancement
New feature or request
labels
Jan 9, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 1, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 1, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 1, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 1, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
It's shaping up nicely, here's what it looks like so far: There is a new variable in zeek.env with the rest of the
If set to The links are (by column):
Things still to work on:
|
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 2, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 5, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 5, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 5, 2024
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 5, 2024
…d_files download page
I've added documentation for the improvements as well. I think I'm satisfied with this feature now, closing. |
This was referenced Feb 14, 2024
Merged
mmguero
added a commit
that referenced
this issue
Feb 15, 2024
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://idaholab.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (#300) - add option for customizing which log types get NetBox enrichment (#316) - improve the extracted_files download page (#329) - include missing aggregations in API bucket queries (#386) - more intelligent .env file checking on startup (#387) - Malcolm report to itself on capture statistics (#395) - link to Dashboards/Arkime from NetBox devices view (#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (#403) - allow netbox-restore and netbox-backup to specify container name (#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (#393) (and [updated documentation](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (#401) - event.severity_tags is not being assigned correctly based on rule.category (#402) - basic authentication breaks with special characters (#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/idaholab/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (#335) and `/dashboard/` redirect (#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/idaholab/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/idaholab/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Feb 15, 2024
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (idaholab#300) - add option for customizing which log types get NetBox enrichment (idaholab#316) - improve the extracted_files download page (idaholab#329) - include missing aggregations in API bucket queries (idaholab#386) - more intelligent .env file checking on startup (idaholab#387) - Malcolm report to itself on capture statistics (idaholab#395) - link to Dashboards/Arkime from NetBox devices view (idaholab#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403) - allow netbox-restore and netbox-backup to specify container name (idaholab#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401) - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402) - basic authentication breaks with special characters (idaholab#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
carving
Relating to carving (extraction) of files from traffic and the scanning of those files
enhancement
New feature or request
Here's what the
/extracted-files/
screen looks like now:We could improve this I think in a few ways:
This page is generated by this python script
The text was updated successfully, but these errors were encountered: