Skip to content

Commit

Permalink
update graduation criteria, version, test plan
Browse files Browse the repository at this point in the history
  • Loading branch information
jmrodri committed Dec 6, 2021
1 parent 653453e commit 9a387c2
Showing 1 changed file with 20 additions and 68 deletions.
88 changes: 20 additions & 68 deletions enhancements/sdk-external-and-pluggable-validations.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,80 +349,32 @@ of compiling them in.

### Test Plan

**Note:** *Section not required until targeted at a release.*

Consider the following in developing a test plan for this enhancement:
- Will there be e2e and integration tests, in addition to unit tests?
- How will it be tested in isolation vs with other components?

No need to outline all of the test cases, just the general strategy. Anything
that would count as tricky in the implementation and anything particularly
challenging to test should be called out.

All code is expected to have adequate tests (eventually with coverage
expectations).
* unit testing of the sdk external validator feature
* bundle validate e2e
* external validators will have their own test coverages

### Graduation Criteria

**Note:** *Section not required until targeted at a release.*

Define graduation milestones.

These may be defined in terms of API maturity, or as something else. Initial proposal
should keep this high-level with a focus on what signals will be looked at to
determine graduation.

Consider the following in developing the graduation criteria for this
enhancement:
- Maturity levels - `Dev Preview`, `Tech Preview`, `GA`
- Deprecation

Clearly define what graduation means.

#### Examples

These are generalized examples to consider, in addition to the aforementioned
[maturity levels][maturity-levels].

##### Dev Preview -> Tech Preview

- Ability to utilize the enhancement end to end
- End user documentation, relative API stability
- Sufficient test coverage
- Gather feedback from users rather than just developers
- dev preview will require the user to specify the location of the validator to
run

##### Tech Preview -> GA

- More testing (upgrade, downgrade, scale)
- Sufficient time for feedback
- Available by default

**For non-optional features moving to GA, the graduation criteria must include
end to end tests.**
- TBD

##### Removing a deprecated feature

- Announce deprecation and support policy of the existing feature
- Deprecate the feature
N/A

### Upgrade / Downgrade Strategy

If applicable, how will the component be upgraded and downgraded? Make sure this
is in the test plan.

Consider the following in developing an upgrade/downgrade strategy for this
enhancement:
- What changes (in invocations, configurations, API use, etc.) is an existing
cluster required to make on upgrade in order to keep previous behavior?
- What changes (in invocations, configurations, API use, etc.) is an existing
cluster required to make on upgrade in order to make use of the enhancement?
N/A

### Version Skew Strategy

* The version of the validators can change however the validator author sees fit.
* The API or contract between `operator-sdk` and validators will be
* input to validator: bundle directory
* output from validator: `ManifestResult` JSON
N/A

## Implementation History

Expand All @@ -436,6 +388,17 @@ compilation step being needed depending on the language used to implement them.

## Alternatives

* use a language like JavaScript or CUE to define all validations
* validations could be run from a git repo, i.e. operator-sdk could pull it
and then evaluate it
* pro:
* simpler delivery, expose via a gitrepo and done
* con:
* all existing validations would have to be re-written in a new language
structure which could introduce new bugs
* unproven technology
* would have to write the engine to know how to execute these

* put validations in their own images
* need to define "API" contract what is the entrypoint and what parameters do
we give it
Expand All @@ -460,17 +423,6 @@ TODO: ---------- vvvvvv CUT vvvvvv ----------
* authors would have to create binaries of their validations
TODO: ---------- ^^^^^^ CUT ^^^^^^ ----------

* use a language like JavaScript or CUE to define all validations
* validations could be run from a git repo, i.e. operator-sdk could pull it
and then evaluate it
* pro:
* simpler delivery, expose via a gitrepo and done
* con:
* all existing validations would have to be re-written in a new language
structure which could introduce new bugs
* unproven technology
* would have to write the engine to know how to execute these

* use scorecard to do the validations
* create validations written in scorecard as custom tests
* pro:
Expand Down

0 comments on commit 9a387c2

Please sign in to comment.