[Security Solution] Prebuilt rule status API endpoint returns incorrect total number of prebuilt rules

[banderror]

Found here: #148392 (comment)
Related to: #148182

Summary

As found by @xcrzx:

There's an error somewhere in the total number of prebuilt rules calculations. I have all rules installed, and one of them is outdated. That's what I get from the status endpoint:

{
    "status_code": 200,
    "message": "OK",
    "attributes": {
        "stats": {
            "num_prebuilt_rules_total": 756,
            "num_prebuilt_rules_installed": 757,
            "num_prebuilt_rules_to_install": 0,
            "num_prebuilt_rules_to_upgrade": 1,
            "rule_ids_to_install": [],
            "rule_ids_to_upgrade": [
                "000047bb-b27a-47ec-8b62-ef1a5d2c9e19"
            ]
        }
    }
}

num_prebuilt_rules_total should be 757.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[xcrzx]

After further investigation, it appears that the total number of prebuilt rules could be less than the number of installed prebuilt rules. This occurs when a newer package has previously deprecated rules removed. For example, here's what I get locally with a fairly old rules package installed:

{
    "status_code": 200,
    "message": "OK",
    "attributes": {
        "stats": {
            "num_prebuilt_rules_total": 679,
            "num_prebuilt_rules_installed": 754,
            "num_prebuilt_rules_to_install": 4,
            "num_prebuilt_rules_to_upgrade": 4
        }
    }
}

So, it is not a bug. Nevertheless, the response fields could be confusing. The num_prebuilt_rules_total naming might imply that this is the total number of prebuilt rules (installed + available), but in fact, it is the number of prebuilt rules in the latest package. Probably, we need to rename the field accordingly.

[xcrzx]

We've discussed this issue during a group sync and decided to remove the num_prebuilt_rules_total for now.