OCPBUGS-9382: Expose validation issue when creating TailoredProfiles

[rhmdnd]

TailoredProfiles can extend existing profiles, which can be either Node
or Platform type.

However, it's possible to create a TailoredProfile that extends a
profile of a patricular type, and then reference rules of the opposite
type. This causes issues during scans because you'd expect the rules to
be excluded, but they're not.

This commit adds an e2e test the exposes the issue. This can be
addressed with improved validation when creating a TailoredProfile.

Related to issue #65.

[openshift-ci-robot]

@rhmdnd: This pull request references Jira Issue OCPBUGS-9382, which is invalid:

• expected the bug to target the "4.14.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

TailoredProfiles can extend existing profiles, which can be either Node
or Platform type.

However, it's possible to create a TailoredProfile that extends a
profile of a patricular type, and then reference rules of the opposite
type. This causes issues during scans because you'd expect the rules to
be excluded, but they're not.

This commit adds an e2e test the exposes the issue. This can be
addressed with improved validation when creating a TailoredProfile.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhmdnd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhmdnd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@rhmdnd: This pull request references Jira Issue OCPBUGS-9382, which is invalid:

• expected the bug to target the "4.14.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

TailoredProfiles can extend existing profiles, which can be either Node
or Platform type.

However, it's possible to create a TailoredProfile that extends a
profile of a patricular type, and then reference rules of the opposite
type. This causes issues during scans because you'd expect the rules to
be excluded, but they're not.

This commit adds an e2e test the exposes the issue. This can be
addressed with improved validation when creating a TailoredProfile.

Related to issue #65.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xiaojiey]

/hold for test

[xiaojiey]

It seems that it is only the tp validation issue. When you execute scan with the tp, the disabled rules will not included:
##################disable node type when extending a platform type profile:
$ oc apply -f -<<EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
name: ocp4-cis-modified
spec:
extends: ocp4-cis # this is a profile dedicated to Platform checks
description: CIS Benchmarks profile
title: Modified CIS NodeBenchmarks profile
disableRules:
- name: ocp4-kubelet-enable-protect-kernel-defaults # this is a Node type rule
rationale: RFE-2714 - This is set by default, no need to adjust kubelet configuration
EOF
tailoredprofile.compliance.openshift.io/ocp4-cis-modified created
$ oc get tp -w
NAME STATE
ocp4-cis-modified READY
^C
$ oc compliance bind -N test tailoredprofile/ocp4-cis-modified
Creating ScanSettingBinding test
$ oc get suite -w
NAME PHASE RESULT
test RUNNING NOT-AVAILABLE
test AGGREGATING NOT-AVAILABLE
test DONE NON-COMPLIANT
test DONE NON-COMPLIANT
^C$ oc get ccr | grep protect-kernel-defaults
$ oc get ccr | grep admin
ocp4-cis-modified-kubeadmin-removed FAIL medium
ocp4-cis-modified-rbac-limit-cluster-admin MANUAL medium

[xiaojiey]

/unhold

[rhmdnd]

E2E test is working as expected.

=== CONT  TestTailoredProfileRuleValidation
2023/04/14 18:59:00 waiting until suite test-suite-with-non-matching-content reaches target status 'DONE'. Current status: RUNNING
2023/04/14 18:59:00 waiting ProfileBundle test-profile-modification to become VALID (PENDING)
2023/04/14 18:59:00 waiting until suite test-scheduled-suite-update reaches target status 'DONE'. Current status: RUNNING
2023/04/14 18:59:00 ProfileBundle ready (VALID)
2023/04/14 18:59:00 waiting until suite test-scheduled-suite-invalid-priority-class reaches target status 'DONE'. Current status: RUNNING
2023/04/14 18:59:00 Waiting for run of test-scan-w-missing-tailoring-cm compliancescan (RUNNING)
2023/04/14 18:59:01 Waiting for run of test-missing-pod-scan compliancescan (RUNNING)
2023/04/14 18:59:01 Object found 'test-tailored-profile-rule-validation' found
    main_test.go:2842: TailoredProfile test-tailored-profile-rule-validation expected to be in error state, but it's actually in READY
--- FAIL: TestTailoredProfileRuleValidation (5.24s)

Just need to find an appropriate place for the validation.

[openshift-ci]

@rhmdnd: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-parallel	1db3eb0	link	true	/test e2e-aws-parallel
ci/prow/e2e-rosa	1db3eb0	link	true	/test e2e-rosa
ci/prow/e2e-aws-serial	1db3eb0	link	true	/test e2e-aws-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.