Wire new Release + SDLC Core Entities #3839

puerco · 2024-07-11T03:10:31Z

Summary

This PR adds new entities to enable us to create new minder integrations into the software development lifecycle. The new entities are named based on the CD Events specification conventions when possible and include:

Release: A software release based on a tagged commit
PipelineRun: An execution of a CICD pipeline, akin to a GitHub workflow
TaskRun: An execution of a task part of a pipeline
Build: A form of task that builds software

This PR adds the new entity types to the protos in ebc10f5, creates stub messages in bba557f and wires them to the Minder machinery.

Storage of the new entities into the database is not yet implemented as the data structures are not yet defined but this PR includes the migrations to add the new entities to the sql type enum in 268e217

Line count is high but most of the changes lines come from code regeneration in f68e844

Note: This PR is intended to be innocuous: No provider is yet reacting or implementing logic to handle any of these new types, it should have no effect. Where possible, the entity types are fully wired in, when not I've added NOOP cases that will be replaced when the data structures are defined

/cc @eleftherias @rdimitrov

Change Type

Mark the type of change your PR introduces:

Bug fix (resolves an issue without affecting existing features)
Feature (adds new functionality without breaking changes)
Breaking change (may impact existing functionalities or require documentation updates)
Documentation (updates or additions to documentation)
Refactoring or test improvements (no bug fixes or new functionality)

Testing

Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.

Review Checklist:

Reviewed my own code for quality and clarity.
Added comments to complex or tricky code sections.
Updated any affected documentation.
Included tests that validate the fix or feature.
Checked that related changes are merged.

coveralls · 2024-07-11T03:19:23Z

coverage: 52.92% (-0.2%) from 53.082%
when pulling 4befe24 on puerco:release-entity
into 0342f5a on stacklok:main.

jhrozek · 2024-07-11T07:52:25Z

database/migrations/000073_sdlc_entities.up.sql

+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+ALTER TYPE entities ADD VALUE 'release';


could we add a transaction here so either the migration proceeds or fails as a whole?

I remember what happened when we hit this bug so this is good feedback 😃

Yeah good call, added it 👍

eleftherias · 2024-07-11T10:01:36Z

@puerco Where do you see us using PipelineRun and TaskRun? It's clear to me where we would use Release and Build (assuming Build is akin to a GitHub workflow file?), but I'm not sure yet where the other 2 fit in.

database/migrations/000073_sdlc_entities.down.sql

rdimitrov · 2024-07-11T14:23:52Z

pkg/api/protobuf/go/minder/v1/entities.go

+	// ReleaseEntity is an entity abstracting a release
+	ReleaseEntity EntityType = "release"
+	// PipelineRunEntity is an entity abstracting a pipeline run (eg a workflow)
+	PipelineRunEntity EntityType = "pipeline_run"
+	// TaskRunEntity is an entity abstracting a task run (eg a step)
+	TaskRunEntity EntityType = "task_run"
+	// BuildEntity is an entity that represents a software build
+	BuildEntity EntityType = "build"


@eleftherias - I think this is where @puerco put more info about which corresponds to what in GitHub terms. In that sense, I guess task_run might be a job instead of a step. And also if pipeline_run is the workflow, then what is the build? But I guess we don't have to define these now, right? We can do that once we start figuring out the specific provider implementation

@eleftherias @rdimitrov yes, with these new events we should be able to write rules against the results of a Workflow (a PipelineRun in cd events terms), a Step (a TaskRun in cd events) and also to published releases.

The build itself in GitHub terms will look like a subclass of a Step run. It will share a lot of attributes but also has some particulars that differ from a regular Step run that we will need to check (dependencies, the builder, etc).

Do you know what kind of rules we want to write against the results of a PipelineRun and TaskRun? I'm not yet seeing how they tie into the new rules we wanted to write and I want to make sure we're not adding entities that will sit around unused.

Perhaps it's difficult to add a specific answer, but my understanding is this should allows us to have more control over the software release lifecycle, i.e. at the end of this (workflow, job, step) we should have that, if not - block executing what's coming next.

I guess another example would be to gather some metadata after each step of the delivery process and store it into an attestation.

rdimitrov

Added a few comments, but apart from the need to rebase I think we can merge it 👍

This commit adds 'release' plus some of the core subjects of the cd events spec as entities to the minder protos: ENTITY_RELEASE ENTITY_PIPELINE_RUN ENTITY_TASK_RUN ENTITY_BUILD Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

This commit adds stub messages for the SDLC entities: Release PipelineRun TaskRun Build Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

This commit adds up and down (although down is empty) migrations to add release and the core cd events subjects to the sql entities type. Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

(Run make gen) Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

This adds release and the core SLDX entity types to the profile functions Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

This commit adds the release and SLDC entities to the control plane functions Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

This commit wires the SDLC entities to the engine functions and creates the With* functional options for the new types. Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

This commit adds NOOP cases to functions where the SDLC entities need to be implemented in the database or, where hardcoded entity types in functions prevent us from wiring up the values. Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

puerco · 2024-07-11T17:55:18Z

OK, I rebased and fixed the wreck from last night's changes. PTAL!

puerco requested a review from a team as a code owner July 11, 2024 03:10

puerco force-pushed the release-entity branch from b05d333 to 26db55d Compare July 11, 2024 03:11

puerco changed the title ~~Wire new release + SDLC Core Entities~~ Wire new Release + SDLC Core Entities Jul 11, 2024

jhrozek reviewed Jul 11, 2024

View reviewed changes

rdimitrov reviewed Jul 11, 2024

View reviewed changes

database/migrations/000073_sdlc_entities.down.sql Outdated Show resolved Hide resolved

rdimitrov reviewed Jul 11, 2024

View reviewed changes

puerco force-pushed the release-entity branch from 26db55d to ba9dc06 Compare July 11, 2024 15:47

puerco added 2 commits July 11, 2024 10:01

Proto: Add SDLC stubs

b5cf2c3

This commit adds stub messages for the SDLC entities: Release PipelineRun TaskRun Build Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

puerco force-pushed the release-entity branch from ba9dc06 to ef1e21e Compare July 11, 2024 16:01

puerco added 8 commits July 11, 2024 11:35

Add SQL migrations for SDLC entities

fccd75a

This commit adds up and down (although down is empty) migrations to add release and the core cd events subjects to the sql entities type. Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

SDLXC entities: Make gen

a77bcf0

(Run make gen) Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

Add SLDC entities to Profile

bfefa8d

This adds release and the core SLDX entity types to the profile functions Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

Add release+SLDC entities to controlplane

388e4b4

This commit adds the release and SLDC entities to the control plane functions Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

Wire SDLC entities to engine

ce30fc3

This commit wires the SDLC entities to the engine functions and creates the With* functional options for the new types. Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

Handle SDLC entities in mindev

f603806

Add SDLC noops to history and exec aggregator

6186169

Signed-off-by: Adolfo García Veytia (Puerco) <puerco@stacklok.com>

puerco force-pushed the release-entity branch from ef1e21e to 6186169 Compare July 11, 2024 17:36

Merge branch 'main' into release-entity

4befe24

rdimitrov approved these changes Jul 12, 2024

View reviewed changes

puerco merged commit 33ae09f into mindersec:main Jul 12, 2024
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wire new Release + SDLC Core Entities #3839

Wire new Release + SDLC Core Entities #3839

puerco commented Jul 11, 2024 •

edited

Loading

coveralls commented Jul 11, 2024 •

edited

Loading

jhrozek Jul 11, 2024

rdimitrov Jul 11, 2024

puerco Jul 11, 2024

eleftherias commented Jul 11, 2024

rdimitrov Jul 11, 2024

puerco Jul 11, 2024

eleftherias Jul 12, 2024

rdimitrov Jul 12, 2024

rdimitrov Jul 12, 2024

rdimitrov left a comment

puerco commented Jul 11, 2024

Wire new Release + SDLC Core Entities #3839

Wire new Release + SDLC Core Entities #3839

Conversation

puerco commented Jul 11, 2024 • edited Loading

Summary

Change Type

Testing

Review Checklist:

coveralls commented Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eleftherias commented Jul 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdimitrov left a comment

Choose a reason for hiding this comment

puerco commented Jul 11, 2024

puerco commented Jul 11, 2024 •

edited

Loading

coveralls commented Jul 11, 2024 •

edited

Loading