Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploy YARA rules remotely and privately #14899

Closed
2 of 15 tasks
zayhanlon opened this issue Nov 2, 2023 · 33 comments
Closed
2 of 15 tasks

Deploy YARA rules remotely and privately #14899

zayhanlon opened this issue Nov 2, 2023 · 33 comments
Assignees
@zwass
Labels
~csa Issue was created by or deemed important by the Customer Solutions Architect. customer-domon customer-seidel #g-orchestration Orchestration product group :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. story A user story defining an entire feature
Milestone
4.61.0

Comments

@zayhanlon
Copy link
Contributor

zayhanlon commented Nov 2, 2023 •
edited by lucasmrod
Loading

User story
As a detection & response engineers,
I want to deploy YARA rules to agents remotely and privately from a server I host myself (separate from Fleet)
so don't have to write rules to disk (too large of scope and too slow) or host rules on a non-private webserver (what osquery supports today).

Context

Why doesn't current YARA rule deployment with osquery work?

  • Too large of scope: If they wrote the rules to disk, any engineering (including outside detection & response team) could read these rules because they'd live in the organization's monorepo. If an engineer's account gets compromised then they can read these rules.
  • Too slow: It takes months to get these rules reviewed.
  • Privately: Viewing rules must require authentication. Currently, osquery doesn't provide a way to authenticate YARA rules.

Changes

Product

  • UI changes: No changes
  • CLI (fleetctl) usage changes: No changes
  • YAML changes: TODO: Specify changes in the YAML files doc page as a PR to the reference docs release branch. Put "No changes" if there are no changes necessary.
  • REST API changes: TODO: Specify changes in the the REST API doc page as a PR to reference docs release branch. Put "No changes" if there are no changes necessary. Move this item to the engineering list below if engineering will design the API changes.
  • Fleet's agent (fleetd) changes: PR to allow osquery to authenticate YARA requests: Add --yara_sigurl_authenticate flag osquery/osquery#8437. With the yara_sigurl_authenticate flag enabled, osquery will send the node key when retrieving YARA rules, which will allow the Fleet server to authenticate the request before responding.
  • Activity changes: No changes
  • Permissions changes: No changes.
  • Changes to paid features or tiers: TODO: Specify changes in pricing-features-table.yml as a PR to reference docs release branch. Specify "Fleet Free" and/or "Fleet Premium" if there are no changes to the pricing page necessary.
  • Once shipped, requester has been notified
  • Once shipped, dogfooding issue has been filed

Engineering

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

  • Requires load testing: TODO
  • Risk level: Low / High TODO
  • Risk description: TODO

Manual testing steps

  1. Step 1
  2. Step 2
  3. Step 3

Testing notes

Confirmation

  1. Engineer (@____): Added comment to user story confirming successful completion of QA.
  2. QA (@____): Added comment to user story confirming successful completion of QA.
@zayhanlon zayhanlon added customer-domon ~customer request A prioritized, customer feature request. Has ≥ 1 customer codename label(s) ~feature fest Will be reviewed at next Feature Fest labels Nov 2, 2023
@noahtalerman
Copy link
Member

Feature fest: We have a yara table in Fleet: https://fleetdm.com/tables/yara

Does that address the need?

@noahtalerman noahtalerman removed the ~feature fest Will be reviewed at next Feature Fest label Nov 2, 2023
@zayhanlon zayhanlon added ~feature fest Will be reviewed at next Feature Fest and removed ~customer request A prioritized, customer feature request. Has ≥ 1 customer codename label(s) labels Dec 7, 2023
@noahtalerman noahtalerman removed the ~feature fest Will be reviewed at next Feature Fest label Dec 15, 2023
@zayhanlon zayhanlon added the ~feature fest Will be reviewed at next Feature Fest label Jun 6, 2024
@zayhanlon
Copy link
Contributor Author

Related to an ask from customer-domon re: #19553

@JoStableford
Copy link
Contributor

Related to a Slack conversation

@nonpunctual nonpunctual added the ~csa Issue was created by or deemed important by the Customer Solutions Architect. label Jun 6, 2024
@nonpunctual
Copy link
Contributor

nonpunctual commented Jun 6, 2024 •
edited
Loading

The yara & yara_events tables do not have the capabilities to access sigrules from a remote server securely.

@noahtalerman noahtalerman changed the title Fleet to set/important YARA rules like osquery query packs 🎸 Fleet to set/important YARA rules like osquery query packs Jun 10, 2024
@noahtalerman noahtalerman mentioned this issue Jun 20, 2024
Closed
@noahtalerman noahtalerman added story A user story defining an entire feature :product Product Design department (shows up on 🦢 Drafting board) and removed ~feature fest Will be reviewed at next Feature Fest labels Jun 21, 2024
@zayhanlon
Copy link
Contributor Author

@noahtalerman let me know if you or design team would like to chat with the customer during or after this air guitar. thanks!

@noahtalerman
Copy link
Member

Hey @zayhanlon! Would love some help setting up a call w/ the customer.

@noahtalerman
Copy link
Member

Containers supported by YARA (Specifically Kubernetes):
Brock: Need context → Adding namespace column by PID would allow you to use YARA rules in containers
kubequery in the repo may be helpful
May be a blocker since they don’t have direct Kubernetes API
Brock: Could pass along to infra team to use
Ben: Potentially, could be hard though
Concern was if they could use osquery to run YARA rules in containers that are Docker → Now use Kubernetes so want that same ability for Kube
osquery does not support containerd → this would resolve the issue
TODO Brock: Bring this to eng to discuss
Found article about osquery support for containerd
https://developer.ibm.com/articles/monitoring-containers-osquery/

@nonpunctual
Copy link
Contributor

Hey @noahtalerman I did reach out to IBM on their GitHub to see if they had open-sourced anything in relation to this. Got no response. That said, I think we could duplicate what was built based on the article.

@noahtalerman noahtalerman removed :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. #g-endpoint-ops Endpoint ops product group labels Dec 11, 2024
@noahtalerman
Copy link
Member

@zwass also, did this get QA'd by one of our QA engineers? If not, can you please add it to the QA column on the #g-endpoint ops board and tag Sharon to let him know?

cc @zayhanlon

@lucasmrod
Copy link
Member

I reviewed the PR and this was shipped in 4.60.0. If we missed QA it was probably because it has :product and doesn't have the :release (and doesn't have a Milestone set).

@noahtalerman
Copy link
Member

If we missed QA it was probably because it has :product and doesn't have the :release (and doesn't have a Milestone set).

I added :product yesterday (after we shipped). Sounds like this one missed QA. We want to make sure the specs in the "Product" section are filled out so QA knows the definition of done.

  • YAML changes: TODO: Specify changes in the YAML files doc page as a PR to the reference docs release branch. Put "No changes" if there are no changes necessary.
  • REST API changes: TODO: Specify changes in the the REST API doc page as a PR to reference docs release branch. Put "No changes" if there are no changes necessary. Move this item to the engineering list below if engineering will design the API changes.
  • Permissions changes: TODO: Specify changes in the Manage access doc page as a PR to the reference docs release branch. If doc changes aren't necessary, explicitly mention no changes to the doc page. Put "No changes" if there are no permissions changes.
  • Changes to paid features or tiers: TODO: Specify changes in pricing-features-table.yml as a PR to reference docs release branch. Specify "Fleet Free" and/or "Fleet Premium" if there are no changes to the pricing page necessary.

@zwass can you please add PRs/specs for these remaining TODOs.

(I think I had removed them we were working on this story. Whoops from me)

When those are spec'd can you please add :release and #g-endpoint-ops labels and ping Sharon that this one needs QA?

cc @zayhanlon

@noahtalerman noahtalerman mentioned this issue Dec 12, 2024
Closed
@zayhanlon
Copy link
Contributor Author

@lucasmrod @noahtalerman -- we merged these docs awhile ago https://github.com/fleetdm/fleet/pull/24015/files but i can't figure out where it is or where it ended up on the website. did we put it in the wrong place?

@lucasmrod
Copy link
Member

@lucasmrod @noahtalerman -- we merged these docs awhile ago https://github.com/fleetdm/fleet/pull/24015/files but i can't figure out where it is or where it ended up on the website. did we put it in the wrong place?

Ah, it was merged to the docs-v4.61.0 branch so the docs will be released when 4.61.0 is released.

@zwass
Copy link
Member

zwass commented Dec 12, 2024

Ah, seems like I may have been confused about when it was going out. Maybe should merge it to main now if it shipped to 4.60? Apologies that I'm not on the ball with these process things.

@zayhanlon zayhanlon added #g-endpoint-ops Endpoint ops product group :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. and removed #g-customer-success Customer success issue. :product Product Design department (shows up on 🦢 Drafting board) labels Dec 12, 2024
@zayhanlon
Copy link
Contributor Author

@sharon-fdm i dont think this went through formal qa last sprint, so we're circling it back for qa in case it was missed per noah's comments above. can you check this out and also review if the docs are in the right place? @xpkoala fyi

@sharon-fdm
Copy link
Collaborator

cc: @xpkoala, @jmwatts.

noahtalerman added a commit that referenced this issue Dec 17, 2024
@noahtalerman
Handbook: make sure changes outside of product groups are QA'd
52ce2d0 
Recent change that missed QA:
- #14899 (comment)
@noahtalerman noahtalerman mentioned this issue Dec 17, 2024
Merged
@lucasmrod
Copy link
Member

@lucasmrod @noahtalerman -- we merged these docs awhile ago https://github.com/fleetdm/fleet/pull/24015/files but i can't figure out where it is or where it ended up on the website. did we put it in the wrong place?

4.61.0 has been released, so now we have the guide published:
https://fleetdm.com/guides/remote-yara-rules

@lucasmrod
Copy link
Member

@zayhanlon ⬆️

@lucasmrod lucasmrod mentioned this issue Dec 20, 2024
Open
3 tasks
@sharon-fdm sharon-fdm added the #g-orchestration Orchestration product group label Jan 2, 2025
@sharon-fdm sharon-fdm modified the milestones: 4.62.0, 4.61.0 Jan 2, 2025
@sgress454 sgress454 mentioned this issue Jan 6, 2025
Merged
@sharon-fdm sharon-fdm removed the #g-endpoint-ops Endpoint ops product group label Jan 6, 2025
noahtalerman pushed a commit that referenced this issue Jan 7, 2025
@sgress454
Add remote YARA language to pricing chart (#25181)
c846e30 
relates to #14899
sgress454 added a commit that referenced this issue Jan 7, 2025
@sgress454
Add remote YARA language to pricing chart (#25181)
ef84035 
relates to #14899
@xpkoala xpkoala closed this as completed Jan 8, 2025
@fleet-release
Copy link
Contributor

Deploying YARA rules,
Privately, swift as a hawk,
No trace left behind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zwass zwass

Labels
~csa Issue was created by or deemed important by the Customer Solutions Architect. customer-domon customer-seidel #g-orchestration Orchestration product group :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. story A user story defining an entire feature
Milestone
4.61.0
Development

No branches or pull requests