Should policy runs for hosts that are not within scope of associated software automation?

[jmwatts]

UPDATE: @noahtalerman: Closed this issue after we decided the following:

• Release as-is with copy updates (Figma here) for 4.62.0. Issue with copy updates is here.
• In a follow up iteration, update the UI so that policies outside software scope are filtered from the list of policies on the host details page. Tracked in a user story here: Hide policies outside software target on Host details and My device page #25226
• In the iteration in which we draft "Scope policies using labels" (Scope policies using labels #24097), Which the #g-orchestration team will take on this quarter (Q1 2025), decide if we want to revise policy behavior or extend our policy results processing and keep policies as-is.

---

Fleet version: v4.62.0

Web browser and operating system: Chrome 131.0.6778.205 on macOS

---

💥  Actual behavior

In Fleet log:
level=debug ts=2024-12-31T17:38:18.781518Z host_id=14 host_platform=darwin policy_id=67 software_installer_id=755 software_title_id=6185 software_installer_platform=darwin msg="not marking policy as failed since software is out of scope for host"

🧑‍💻  Steps to reproduce

1. Add a label to an existing macOS host (ex: Testing scope)
2. Go to Software >> Add software
3. Choose either Fleet-maintained or Custom package
4. Choose "Automatic" for "Install" method
5. Choose "Custom" for "Target" and select "Exclude any"
6. Select label from step 1
7. Click Add software
8. On the host from step 1, open the My device page >> Policies and look for policy created to install the software just added
9. On My device page and click Refetch
10. Watch the Fleet log

🕯️ Expected behavior

Per the design for this feature, the policy "won't be checked on hosts that aren't in the scope" but the policy does run - it just doesn't trigger installation of the software and also doesn't record the result of the policy being run.

I would expect the policy to not show up in My device and for it not to run at all based on the design for this feature.

[lukeheath]

@noahtalerman @marko-lisica This was originally filed as a bug since it did not match the design spec of not running policies at all.

Not running the policy at all is more difficult to implement, so the decision was made to release as-is with the policy running but returning --- if the software was not within scope. I'm re-purposing this to a design ticket for us to consider the next iteration.

Some concerns expressed:

1. Currently, policies run on all hosts within their target. This is easy to reason about.
2. With this change, policies would not run on all hosts in their target, but instead would be restricted by another entity type (software) that is managed in a difference place. This is difficult to reason about (for both the engineer and our users).

• Maybe we actually want the current implementation as it maintains the "Policies run on all hosts in their target" rule.

[iansltx]

Just had similar concerns in my conversation with @jahzielv re: #25071. Agreed with @lukeheath that "whether a policy runs" should be distinct from "whether the automation for the policy is in scope." We already see this for edge cases on policy script automations: if you supply a *nix shell script and your policy includes Windows hosts, a Windows host can fail the policy but the shell script won't run. Which winds up being basically this workflow:

1. If the policy query returns zero rows, fail the policy
2. If the policy fails and the host is able to run the script, run the script
3. If the host has scripts enabled and the OS matches the script type, the host can run the script

This is congruent with the workflow for software installers, with the caveat around marking policy failures as blank:

1. If the policy query returns zero rows, fail the policy
2. If the policy fails and the host is able to install the software, install the software
3. If the host is in-scope for the installer and matches the installer target platform, the host can install the software

My concern here is that we're coupling software install targeting with policies anyway with the "blank out policy status if a host fails and has a software automation but that automation is not in scope" logic. This means that policies with a software automation that's out-of-scope have two statuses rather than three: "Success" and "blank", which has the side effect of making #25071 more of an edge case.

This means that the "right way" for solving this is to implement #24097, plus the ability to ensure labels stay in sync between a policy and the associated installer, rather than something to the effect of #24533. At which point (maybe before) we can remove the "whether an installer is in scope influences a policy's status" hack.

Let me know if I'm off track here.

[lukeheath]

I relabeled this as a story for the time being until I've had the chance to align with @noahtalerman on expected behavior.

[noahtalerman]

Luke: We're evaluating all hosts. When we say "evaluating" does that mean the policy's query is run on the hosts?

Noah: That's what we want to avoid. We want to only run the policy's query if it's a member of the software's labels. Later, we want to add the ability to add target policies to only run on specific hosts based on labels (feature request here). Only run the policy and display the result if it's a member of specific labels.

[lukeheath]

From our call today, the plan is:

• Release as-is for 4.62.0.
• Update so that policies outside software scope are filtered from the list of policies on the host details page.
• In future iteration, consider if we want to revise policy behavior or extend our policy results processing and keep policies as-is.

[noahtalerman]

Follow up to Luke's comment about the plan here:

Release as-is for 4.62.0.

As-is with copy updates. I updated the copy in the UI so the IT admin knows why some hosts return an empty (---) status.

@lukeheath is it possible to get this copy update in before the 4.62 release?

Figma is here:

Update so that policies outside software scope are filtered from the list of policies on the host details page.

Here's the user story for this:

• Hide policies outside software target on Host details and My device page #25226

Bringing it into the current design sprint. Target is 4.64.

In future iteration, consider if we want to revise policy behavior or extend our policy results processing and keep policies as-is.

Specifically in the iteration in which we draft "Scope policies using labels" (#24097). Which the #g-orchestration team will take on this quarter (Q1 2025) FYI @rachaelshaw

[noahtalerman]

Release as-is for 4.62.0.

As-is with copy updates. I updated the copy in the UI so the IT admin knows why some hosts return an empty (---) status.

@lukeheath is it possible to get this copy update in before the 4.62 release?

Figma is here:

UPDATE: PR with copy update is merged!

[noahtalerman]

Closing this issue after we decided the following:

• Release as-is with copy updates (Figma here) for 4.62.0. Issue with copy updates is here.
• In a follow up iteration, update the UI so that policies outside software scope are filtered from the list of policies on the host details page. Tracked in a user story here: Hide policies outside software target on Host details and My device page #25226
• In the iteration in which we draft "Scope policies using labels" (Scope policies using labels #24097), Which the #g-orchestration team will take on this quarter (Q1 2025), decide if we want to revise policy behavior or extend our policy results processing and keep policies as-is.

cc @jmwatts @lukeheath @marko-lisica

[fleet-release]

In cloud city's glow,
Policies scope to software,
Efficiency does grow.