Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(common): json schema linting for common validation(s) #473

Conversation

mike-winberry
Copy link
Contributor

@mike-winberry mike-winberry commented Jun 10, 2024

Description

  • create json schema for common.Validation.
  • create methods to run linting against a validation
  • add to workflow so that validations are linted when being converted to LulaValidation
  • update docs with validation info and how to add validation linting to vscode
  • create lula dev lint command.
  • update contributing/dev guidelines for updating the schema
    • add to pr template (checklist item)

Related Issue

#364

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Other (security config, docs update, etc)

Checklist before merging

Copy link

netlify bot commented Jun 10, 2024

Deploy Preview for lula-docs canceled.

Name Link
🔨 Latest commit 500574f
🔍 Latest deploy log https://app.netlify.com/sites/lula-docs/deploys/668ebc3968667e000884ef3b

…d-include-in-output-for-rationale-on-satisfaction-decision
@mike-winberry
Copy link
Contributor Author

mike-winberry commented Jun 13, 2024

  • fix schemas for e2e

mike-winberry and others added 7 commits June 13, 2024 22:47
…d-include-in-output-for-rationale-on-satisfaction-decision
…d-include-in-output-for-rationale-on-satisfaction-decision
…and namespace required if name, none of our test validations follow this rule
…their type enum

chore(makefile): add test-unit to makefile
chore(adr): update validation artifact format (resource-rule) required fields to match current usage and functionality
chore: update domains and provider types with comments from adr, docs, and schema
adr/0007-validation-artifact-format.md Outdated Show resolved Hide resolved
docs/reference/domains/kubernetes-domain.md Outdated Show resolved Hide resolved
src/cmd/tools/compose_test.go Outdated Show resolved Hide resolved
src/pkg/common/oscal/component_test.go Outdated Show resolved Hide resolved
src/pkg/common/schemas/schema.go Outdated Show resolved Hide resolved
src/pkg/common/schemas/schema.go Outdated Show resolved Hide resolved
src/pkg/common/types.go Outdated Show resolved Hide resolved
@meganwolf0
Copy link
Collaborator

I tried this with a lula t lint -f component-1.yaml,component-2.yaml and component-1.yaml was a bad component-definition, but instead of it telling me it was a bad one/it failed linting, it instead said "Failed to compose component-1.yaml". That might be an ok enough error or we might want to sneak an extra component-definition lint in there to get more info? Also, it just stopped there instead of also linting the second file, which I thought we tried to fix in a separate issue so this compose logic might break that somehow.

@mike-winberry mike-winberry marked this pull request as draft July 10, 2024 21:05
@mike-winberry
Copy link
Contributor Author

mike-winberry commented Jul 10, 2024

I tried this with a lula t lint -f component-1.yaml,component-2.yaml and component-1.yaml was a bad component-definition, but instead of it telling me it was a bad one/it failed linting, it instead said "Failed to compose component-1.yaml". That might be an ok enough error or we might want to sneak an extra component-definition lint in there to get more info? Also, it just stopped there instead of also linting the second file, which I thought we tried to fix in a separate issue so this compose logic might break that somehow.

Based on group discussion:

  • remove compose logic from lint
  • create dev lint
  • still in validate
  • still in compose (for now)
  • still in dev validate

@mike-winberry mike-winberry marked this pull request as ready for review July 11, 2024 18:49
…d-include-in-output-for-rationale-on-satisfaction-decision
@meganwolf0
Copy link
Collaborator

tried out the dev lint and I hate to nitpick but would it be possible to get the actual error output instead of just the file failing lint? It looks like that might be a little tricky since you're handling a list of files so if we want to punt this to a separate issue and tackle later I'm good with that. I think it would just be nice to indicate what failed there.

@mike-winberry
Copy link
Contributor Author

tried out the dev lint and I hate to nitpick but would it be possible to get the actual error output instead of just the file failing lint? It looks like that might be a little tricky since you're handling a list of files so if we want to punt this to a separate issue and tackle later I'm good with that. I think it would just be nice to indicate what failed there.

Ill take a look, I think that should be easy ish, I have an idea, ill let you know when its pushed, and you can tell me if it works!

@mike-winberry
Copy link
Contributor Author

mike-winberry commented Jul 12, 2024

tried out the dev lint and I hate to nitpick but would it be possible to get the actual error output instead of just the file failing lint? It looks like that might be a little tricky since you're handling a list of files so if we want to punt this to a separate issue and tackle later I'm good with that. I think it would just be nice to indicate what failed there.

Ill take a look, I think that should be easy ish, I have an idea, ill let you know when its pushed, and you can tell me if it works!

Screenshot 2024-07-11 at 7 27 23 PM

Copy link
Member

@brandtkeller brandtkeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate how this feeds errors down into the assessment result during lula validate. lula dev lint feels like the right placement.

…d-include-in-output-for-rationale-on-satisfaction-decision
@brandtkeller brandtkeller merged commit 23a45b6 into main Jul 12, 2024
4 checks passed
@brandtkeller brandtkeller deleted the 364-check-if-lula-validation-is-malformed-and-include-in-output-for-rationale-on-satisfaction-decision branch July 12, 2024 20:48
This was referenced Jul 12, 2024
mjnagel referenced this pull request in defenseunicorns/uds-core Jul 26, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/setup-node](https://github.com/actions/setup-node) | action
| patch | `v4.0.2` -> `v4.0.3` |
|
[chainguard-dev/setup-chainctl](https://github.com/chainguard-dev/setup-chainctl)
| action | patch | `v0.2.1` -> `v0.2.2` |
| [defenseunicorns/lula](https://github.com/defenseunicorns/lula) | |
patch | `v0.4.1` -> `v0.4.3` |
| defenseunicorns/lula-action | action | digest | `095636b` -> `939e0a3`
|

---

### Release Notes

<details>
<summary>actions/setup-node (actions/setup-node)</summary>

###
[`v4.0.3`](https://github.com/actions/setup-node/compare/v4.0.2...v4.0.3)

[Compare
Source](https://github.com/actions/setup-node/compare/v4.0.2...v4.0.3)

</details>

<details>
<summary>chainguard-dev/setup-chainctl
(chainguard-dev/setup-chainctl)</summary>

###
[`v0.2.2`](https://github.com/chainguard-dev/setup-chainctl/releases/tag/v0.2.2)

[Compare
Source](https://github.com/chainguard-dev/setup-chainctl/compare/v0.2.1...v0.2.2)

#### What's Changed

- using env vars to avoid injection by
[@&#8203;cpanato](https://github.com/cpanato) in
[https://github.com/chainguard-dev/setup-chainctl/pull/9](https://github.com/chainguard-dev/setup-chainctl/pull/9)
- add `config-path` input to configure chainctl for different
environments by [@&#8203;cmdpdx](https://github.com/cmdpdx) in
[https://github.com/chainguard-dev/setup-chainctl/pull/11](https://github.com/chainguard-dev/setup-chainctl/pull/11)

#### New Contributors

- [@&#8203;cpanato](https://github.com/cpanato) made their first
contribution in
[https://github.com/chainguard-dev/setup-chainctl/pull/9](https://github.com/chainguard-dev/setup-chainctl/pull/9)

**Full Changelog**:
chainguard-dev/setup-chainctl@v0.2.1...v0.2.2

</details>

<details>
<summary>defenseunicorns/lula (defenseunicorns/lula)</summary>

###
[`v0.4.3`](https://github.com/defenseunicorns/lula/releases/tag/v0.4.3)

[Compare
Source](https://github.com/defenseunicorns/lula/compare/v0.4.2...v0.4.3)

This release contains a small list of features, fixes, and dependency
updates. The main feature in this release is the inclusion of the JSON
schema for a Lula Validation. This is schema can be used in developing
validations, as the schema can be imported into the IDE of choice (from
`/src/pkg/common/schemas/validation.json`). The`lula dev lint` command
can be used to lint a validation manifest. Additionally, improperly
structured validations will be identified in `lula validate` and `lula
tools compose`.

Some docs updates and additions are also part of this release in support
of the impending rollout of the Lula website.

##### Features

- **common:** json schema linting for common validation(s)
([#&#8203;473](https://github.com/defenseunicorns/lula/issues/473))
([23a45b6](https://github.com/defenseunicorns/lula/commit/23a45b696a3c24653ad2001dc4b883f40e9685c1))

##### Bug Fixes

- **release:** add option to milestone for release process
([#&#8203;535](https://github.com/defenseunicorns/lula/issues/535))
([6fe64d8](https://github.com/defenseunicorns/lula/commit/6fe64d82ac4950214749b5f49a1ada12f43d193a))
- **test:** updated uuid in kyverno validation
([#&#8203;539](https://github.com/defenseunicorns/lula/issues/539))
([81446d9](https://github.com/defenseunicorns/lula/commit/81446d9441e1f062c57fa922e7d3cca833cbfd3e))

##### Miscellaneous

- **deps:** update anchore/sbom-action action to v0.17.0
([#&#8203;541](https://github.com/defenseunicorns/lula/issues/541))
([7c29fb7](https://github.com/defenseunicorns/lula/commit/7c29fb7dbbab163c648b4c04c89a1568206b8407))
- **deps:** update github/codeql-action action to v3.25.13
([#&#8203;507](https://github.com/defenseunicorns/lula/issues/507))
([dc6cb88](https://github.com/defenseunicorns/lula/commit/dc6cb88eb8cda95c4f000988fc88e7ff1493d3cb))
- **deps:** update kubernetes packages to v0.30.3
([#&#8203;543](https://github.com/defenseunicorns/lula/issues/543))
([1bdefce](https://github.com/defenseunicorns/lula/commit/1bdefce3f3e2af86f985f5b5e95d8d5f2c0c3c39))
- **docs:** initial docs structure/changes for feedback
([#&#8203;524](https://github.com/defenseunicorns/lula/issues/524))
([c276fdd](https://github.com/defenseunicorns/lula/commit/c276fdd3d390719e0a7825e0aabcdc50f0c33a0a))

#### What's Changed

- feat(common): json schema linting for common validation(s) by
[@&#8203;mike-winberry](https://github.com/mike-winberry) in
[https://github.com/defenseunicorns/lula/pull/473](https://github.com/defenseunicorns/lula/pull/473)
- fix(test): updated uuid in kyverno validation by
[@&#8203;CloudBeard](https://github.com/CloudBeard) in
[https://github.com/defenseunicorns/lula/pull/539](https://github.com/defenseunicorns/lula/pull/539)
- chore(docs): initial docs structure/changes for feedback by
[@&#8203;meganwolf0](https://github.com/meganwolf0) in
[https://github.com/defenseunicorns/lula/pull/524](https://github.com/defenseunicorns/lula/pull/524)
- fix(release): add option to milestone for release process by
[@&#8203;brandtkeller](https://github.com/brandtkeller) in
[https://github.com/defenseunicorns/lula/pull/535](https://github.com/defenseunicorns/lula/pull/535)
- chore(deps): update kubernetes packages to v0.30.3 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/543](https://github.com/defenseunicorns/lula/pull/543)
- chore(deps): update anchore/sbom-action action to v0.17.0 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/541](https://github.com/defenseunicorns/lula/pull/541)
- chore(deps): update github/codeql-action action to v3.25.13 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/507](https://github.com/defenseunicorns/lula/pull/507)
- chore(main): release 0.4.3 by
[@&#8203;github-actions](https://github.com/github-actions) in
[https://github.com/defenseunicorns/lula/pull/534](https://github.com/defenseunicorns/lula/pull/534)

**Full Changelog**:
defenseunicorns/lula@v0.4.2...v0.4.3

###
[`v0.4.2`](https://github.com/defenseunicorns/lula/releases/tag/v0.4.2)

[Compare
Source](https://github.com/defenseunicorns/lula/compare/v0.4.1...v0.4.2)

This release fixes several bugs with the evaluate, generate, and upgrade
commands, and includes several dependency updates.

Evaluate now allows for setting the threshold property to true for
instances where you're running evaluate against an assessment-result
that only has one result.

Generate for component-definitions now adds a text block within the
component's remark field to describe how to reproduce the generation of
the component.

Upgrade now handles errors for instances when the file selected does not
exist. It also has the input file flag as a requirement.

Smaller changes of note, the lula.dev website codebase has been removed
from the Lula repo into a dedicated repo.

##### Bug Fixes

- **evaluate:** set threshold on single result evaluation
([#&#8203;519](https://github.com/defenseunicorns/lula/issues/519))
([9424ec5](https://github.com/defenseunicorns/lula/commit/9424ec521f1ee1f4ddceb3350f22d4b3edea226d))
- **generate:** create annotation in remarks for how to reproduce the
generation of a component
([#&#8203;520](https://github.com/defenseunicorns/lula/issues/520))
([6b59daf](https://github.com/defenseunicorns/lula/commit/6b59daffea89c82cd1b9b418f9b87cac81a3970e))
- **upgrade:** error handling for non-existent oscal
([#&#8203;529](https://github.com/defenseunicorns/lula/issues/529))
([58c03d5](https://github.com/defenseunicorns/lula/commit/58c03d528f05b42f98b67d7ba73d0ec86b3e5c9a))

##### Miscellaneous

- **deps:** update actions/download-artifact action to v4.1.8
([#&#8203;522](https://github.com/defenseunicorns/lula/issues/522))
([f628db8](https://github.com/defenseunicorns/lula/commit/f628db8a1df82d4357f289a77132839375b69df3))
- **deps:** update actions/setup-go action to v5.0.2
([#&#8203;530](https://github.com/defenseunicorns/lula/issues/530))
([d6fa46e](https://github.com/defenseunicorns/lula/commit/d6fa46ef50d502ec1168282807ead1f4ea02c405))
- **deps:** update actions/setup-node action to v4.0.3
([#&#8203;526](https://github.com/defenseunicorns/lula/issues/526))
([de146f7](https://github.com/defenseunicorns/lula/commit/de146f7283c94bce50bcf7f2492af8615dd1e523))
- **deps:** update actions/upload-artifact action to v4.3.4
([#&#8203;523](https://github.com/defenseunicorns/lula/issues/523))
([1d2334b](https://github.com/defenseunicorns/lula/commit/1d2334b0ff676c32a5e3905db6e184d58872b5b7))
- **deps:** update anchore/sbom-action action to v0.16.1
([#&#8203;528](https://github.com/defenseunicorns/lula/issues/528))
([ebdf05c](https://github.com/defenseunicorns/lula/commit/ebdf05caef149a0e21279e942169b96c4c883713))
- **website:** website removal
([#&#8203;525](https://github.com/defenseunicorns/lula/issues/525))
([575044c](https://github.com/defenseunicorns/lula/commit/575044c5c2b366ee160a2eb477a16a95192bc4e0))

#### What's Changed

- fix(evaluate): set threshold on single result evaluation by
[@&#8203;brandtkeller](https://github.com/brandtkeller) in
[https://github.com/defenseunicorns/lula/pull/519](https://github.com/defenseunicorns/lula/pull/519)
- fix(generate): reproducing a generation annotation by
[@&#8203;brandtkeller](https://github.com/brandtkeller) in
[https://github.com/defenseunicorns/lula/pull/520](https://github.com/defenseunicorns/lula/pull/520)
- fix(upgrade): error handling for non-existent oscal by
[@&#8203;brandtkeller](https://github.com/brandtkeller) in
[https://github.com/defenseunicorns/lula/pull/529](https://github.com/defenseunicorns/lula/pull/529)
- chore(deps): update actions/download-artifact action to v4.1.8 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/522](https://github.com/defenseunicorns/lula/pull/522)
- chore(deps): update actions/upload-artifact action to v4.3.4 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/523](https://github.com/defenseunicorns/lula/pull/523)
- chore(deps): update actions/setup-node action to v4.0.3 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/526](https://github.com/defenseunicorns/lula/pull/526)
- chore(deps): update actions/setup-go action to v5.0.2 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/530](https://github.com/defenseunicorns/lula/pull/530)
- chore(deps): update anchore/sbom-action action to v0.16.1 by
[@&#8203;renovate](https://github.com/renovate) in
[https://github.com/defenseunicorns/lula/pull/528](https://github.com/defenseunicorns/lula/pull/528)
- chore(website): website removal by
[@&#8203;brandtkeller](https://github.com/brandtkeller) in
[https://github.com/defenseunicorns/lula/pull/525](https://github.com/defenseunicorns/lula/pull/525)
- chore(main): release 0.4.2 by
[@&#8203;github-actions](https://github.com/github-actions) in
[https://github.com/defenseunicorns/lula/pull/512](https://github.com/defenseunicorns/lula/pull/512)

**Full Changelog**:
defenseunicorns/lula@v0.4.1...v0.4.2

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config help](https://github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View the
[repository job
log](https://developer.mend.io/github/defenseunicorns/uds-core).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQzOC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Micah Nagel <micah.nagel@defenseunicorns.com>
This was referenced Aug 5, 2024
This was referenced Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

Check if Lula Validation is malformed and include in output for rationale on satisfaction decision
3 participants