Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adopt reconciler logic for orchestrator #320

Merged
merged 21 commits into from
Jun 6, 2023
Merged

Conversation

chrisgacsal
Copy link
Contributor

@chrisgacsal chrisgacsal commented May 24, 2023

Description

The main purpose for this PR to adopt the reconciler (from the runtime_scan/orchestrator/common package)
based workflow for managing API objects such as ScanConfig, Scan, Target, ScanResult.
This also required to refactor the Provider as all operations must be idempotent as the Scan life-cycle is broken up
into multiple stages where controllers handle a single transition state during a reconciling event.

Orchestrator (and Scanner)

The Orchestrator is responsible for managing controllers for each API object type: ScanConfig, Scan, ScanResult.

ScanConfigWatcher

The ScanConfigWatcher is responsible for scheduling/handling Scans based on ScanConfig.
When the ScanConfig has both the Schedule.operationTime and Schedule.CronLine defined and the latter represents
a single point in time (Quartz), the latter is used for defining the actual operationTime for new Scans.

ScanWatcher

The ScanWatcher periodically polls the API to detect if the Scan objects are scheduled to be run. In case there is
new Scan scheduled, it invokes the Provider to get the Targets in scope. When Targets are available it creates
ScanResult object for each. It monitors in-progress Scans and updates their Summary based on the information
gathered from the corresponding ScanResult objects.

ScanResultWatcher

The ScanResultWatcher monitors the API to find ScanResult objects which are in transient state based on
their status.General.State and resourceCleanup fields. It is responsible for invoking RunTargetScan method of the
Provider in order to set up the scanner instance and start the scan job. The ScanResult is marked as DONE with errors
(it is considered failed) if the Provider was unable to set up the scanner instance due to a permanent error.
It also does perform cleanup of the resources created for scan job by invoking the RemoveTargetScan method of the provider
according to the DeleteJobPolicy configuration.

Provider

Interface

The Provider interface has been split into two (Discoverer and Scanner) in order to separate responsibilities for
discovering scan scopes and actually performing scans.

Both Discoverer and Scanner interfaces have been reworked in order to have better separation of concerns between
the Provider and the caller (Orchestrator), like leaking provider specific data back to the caller.

The new Scanner interface also requires any implementation to be idempotent.

VMClarity-Scanner (aka cli)

The vmclarity-scanner does not have timeout for waiting for volume attachment anymore, instead it monitors
the corresponding ScanResult and raises an error if the state for ScanResult is changed to DONE meaning that
scanning cannot be performed due to external (mostly due to provider) error.

Bugfixes/Improvements

  • fix missing /dev mount for vmclarity-scanner container which prevented it to detect when the scanner volume is mounted
  • make provider/aws implementation conform with the new Provider interface including the requirement for idempotency

Issues

Type of Change

[x] Bug Fix
[x] New Feature
[ ] Breaking Change
[x] Refactor
[ ] Documentation
[ ] Other (please describe)

Checklist

  • I have read the contributing guidelines
  • Existing issues have been referenced (where applicable)
  • I have verified this change is not present in other open pull requests
  • Functionality is documented
  • All code style checks pass
  • New code contribution is covered by automated tests
  • All new and existing tests pass

@chrisgacsal chrisgacsal force-pushed the refactor-scan-mgmt branch 30 times, most recently from 6c93b47 to 6fdec0b Compare May 30, 2023 11:00
@chrisgacsal chrisgacsal changed the title refactor: adopt reconciler logic for orchestrator Adopt reconciler logic for orchestrator Jun 5, 2023
@chrisgacsal chrisgacsal self-assigned this Jun 5, 2023
ghost
ghost previously approved these changes Jun 5, 2023
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, all comments as minor that can be addressed in follow ups. I think the AWS provider can probably be cleaned up a lot now that the interface is simpler (remove all the wrapping interfaces for instance/snapshot etc)

cli/pkg/state/vmclarity.go Show resolved Hide resolved
installation/aws/VmClarity.cfn Show resolved Hide resolved
installation/aws/VmClarity.cfn Outdated Show resolved Hide resolved
runtime_scan/pkg/provider/aws/client.go Outdated Show resolved Hide resolved
runtime_scan/pkg/provider/aws/volume.go Outdated Show resolved Hide resolved
shared/pkg/utils/errors.go Outdated Show resolved Hide resolved
@chrisgacsal chrisgacsal requested a review from a user June 5, 2023 16:19
@chrisgacsal chrisgacsal requested a review from FrimIdan June 6, 2023 08:26
@chrisgacsal chrisgacsal requested a review from FrimIdan June 6, 2023 10:01
@chrisgacsal chrisgacsal added this pull request to the merge queue Jun 6, 2023
Merged via the queue into main with commit 8c87e62 Jun 6, 2023
5 checks passed
@chrisgacsal chrisgacsal deleted the refactor-scan-mgmt branch June 6, 2023 10:39
@chrisgacsal chrisgacsal mentioned this pull request Oct 22, 2023
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants