Add enhancement last-scanned-timestamps

[SimonBaeumer]

This PR adds an enhancement to add a field LastScannedTimestamp to the CheckResult CR.

This enhancement is planned to be implemented by myself. This first iteration is to get consensus on the feature request and discuss possible implementations if necessary.

[openshift-ci]

Hi @SimonBaeumer. Thanks for your PR.

I'm waiting for a ComplianceAsCode member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: SimonBaeumer
Once this PR has been reviewed and has the lgtm label, please assign mrogers950 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rhmdnd]

/ok-to-test

[rhmdnd]

I just realized we have a similar field already on the ComplianceSuite and ComplianceScan objects:

╭─lbragstad@p1 ~
╰─➤  $ oc get scans -o json ocp4-cis | jq .status.conditions
[
  {
    "lastTransitionTime": "2024-04-12T20:21:52Z",
    "message": "Compliance scan run is done running the scans",
    "reason": "NotRunning",
    "status": "False",
    "type": "Processing"
  },
  {
    "lastTransitionTime": "2024-04-12T20:21:52Z",
    "message": "Compliance scan run is done and has results",
    "reason": "Done",
    "status": "True",
    "type": "Ready"
  }
]
╭─lbragstad@p1 ~
╰─➤  $ oc get suites -o json cis | jq .status.conditions
[
  {
    "lastTransitionTime": "2024-04-12T20:21:52Z",
    "message": "Compliance suite run is done running the scans",
    "reason": "NotRunning",
    "status": "False",
    "type": "Processing"
  },
  {
    "lastTransitionTime": "2024-04-12T20:21:52Z",
    "message": "Compliance suite run is done and has results",
    "reason": "Done",
    "status": "True",
    "type": "Ready"
  }
]

To keep things consistent, I wonder if we can just implement status conditions on ComplianceCheckResults.

[rhmdnd]

I think the part that makes this tricky is that the data is only flowing one way (from sensor to central). Without a definitive signal that tells central "yep, you have everything" it's required to make an assumption (e.g., I've waited 5 minutes without anything new coming in from this particular suite, which is marked as DONE, I should be safe to build a report?).

[SimonBaeumer]

We could also implement like this:
After a scan report is triggered, wait until ComplianceSuite is in state DONE, than read all results and check that the lastScannedTimestamp is after the last triggered scan time.
If not, retry 10 minutes later.
Would require a small go routine background job though to manage this.

@rhmdnd How likely do you even think it is that a result is send after the suite reports DONE?
Imho we could also implement a small buffer of some minutes after which the the report is sent.

[dashrews78]

Not just a report problem. ACS simply has no way to know if the results match an iteration of a scan.

[SimonBaeumer]

Not just a report problem. ACS simply has no way to know if the results match an iteration of a scan.

Is a guarantee necessary or the current implementation "good enough"?

If a guarantee is necessary we would need some kind of a UUID per CO scan.

[rhmdnd]

What if the ComplianceScan object contained a counter with the number of ComplianceCheckResults it generated the most recent run? Then consumers would have a known quantity to check for.

[SimonBaeumer]

@rhmdnd Is the ComplianceScan object aware of the total check results it creates?

[rhmdnd]

As an integer, it isn't. We could implement that though in the Reconcile loop of the ComplianceScan and tack that number onto the scan.

[xiaojiey]

/hold for test

[SimonBaeumer]

See #512

[openshift-ci]

@SimonBaeumer: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-serial	fec136e	link	true	/test e2e-aws-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[Vincent056]

we have implmentation in #591