[Epic] Dogfood Pipeline

[bobcatfish]

This epic is about using the Pipeline project to test the Pipeline project (so that we can be our own customers and feel their pain :D).

Expected behavior

The Pipeline project should be tested using the Pipeline project itself, e.g. all testing, building, releasing should be done via Pipelines + Tasks.

Current behavior

The Pipeline project is tested, built and released using Prow jobs which trigger scripts.

Additional info

Once we're ready to take this on we'll probably break this out into separate epics, for example:

• Getting something to trigger our Pipelines (Prow hopefully!)
• Pipeline for releasing
• Pipeline for presubmit building + testing

This will probably become a milestone in the new year.

This is a zenhub epic.

[abayer]

Fwiw, we’re planning to work on getting Prow to support running Pipelines some time in the not too distant future. Cc @rawlingsj

[rawlingsj]

Absolutely, we'd love to help in any way we can. Right now Jenkins X builds Jenkins X with Jenkins X, using prow and Knative Build. We're looking to switch to Pipeline as soon as we can so definitely would like to help / contribute to this.

[bobcatfish]

So cool @abayer + @rawlingsj sounds super exciting!!! 🎉

[vincent-pli]

@bobcatfish
Sorry in advance, maybe this is a stupid question 😸
So, we have an existing k8s cluster with tekton to run the build?
What's the version this Tekton? will upgrade it regular?

[bobcatfish]

Those are good questions @vincent-pli !

So, we have an existing k8s cluster with tekton to run the build?

Yes, we'd need to have tekton deployed in the CI cluster, at least initially it'd be the cluster we use to run Prow

What's the version this Tekton? will upgrade it regular?

I'm not 100% sure! I think at least initially we would pin to whatever the most recent release was, and maybe the process would be to upgrade our own cluster whenever we publish a new release (maybe even as a sanity check before announcing it). Or we could use the nightly build, or we could even use HEAD.

So options:

1. Use most recent released version
2. Upgrade every night to nightly build
3. Update from HEAD on every merge

[afrittoli]

Those are good questions @vincent-pli !

So, we have an existing k8s cluster with tekton to run the build?

Yes, we'd need to have tekton deployed in the CI cluster, at least initially it'd be the cluster we use to run Prow

What's the version this Tekton? will upgrade it regular?

I'm not 100% sure! I think at least initially we would pin to whatever the most recent release was, and maybe the process would be to upgrade our own cluster whenever we publish a new release (maybe even as a sanity check before announcing it). Or we could use the nightly build, or we could even use HEAD.

So options:

1. Use most recent released version

2. Upgrade every night to nightly build

3. Update from HEAD on every merge

I'm not sure option (3) will be possible right from the beginning, but as soon as we sorted out other issues I would start using HEAD! If we by chance we merge a breaking change, we'll break our CI as well and thus we'll stop merging changes until that's fixed. As long as the volume of PRs is similar to what we have today we should be able to afford that.

Besides doing CD with a project from the CDF is pretty cool :)

[afrittoli]

Do we have some kind of design document / diagram that explains what kind of setup we're aiming for? Do you guys @bobcatfish @vdemeester need any help in this area? I would be happy to help.

[bobcatfish]

Do we have some kind of design document / diagram that explains what kind of setup we're aiming for? Do you guys @bobcatfish @vdemeester need any help in this area? I would be happy to help.

There's no doc that I know of! @vdemeester has been doing most of the actual work but afaik so far we've been kind of just taking it one step at a time without a clear overall picture. If you want to start putting together an overall plan and weighing the options @afrittoli that would be super helpful 🙏❤️🙏

I also just sent you an invite to our zenhub project in case you havent seen it yet, that'll let you view the other issues linked to this "epic" (none of which deal with what you're proposing tho).

[afrittoli]

Do we have some kind of design document / diagram that explains what kind of setup we're aiming for? Do you guys @bobcatfish @vdemeester need any help in this area? I would be happy to help.

There's no doc that I know of! @vdemeester has been doing most of the actual work but afaik so far we've been kind of just taking it one step at a time without a clear overall picture. If you want to start putting together an overall plan and weighing the options @afrittoli that would be super helpful 🙏❤️🙏

I also just sent you an invite to our zenhub project in case you havent seen it yet, that'll let you view the other issues linked to this "epic" (none of which deal with what you're proposing tho).

Thank you! I've been ramping up on the different bits and pieces, I'll start working on some document soon.

[bobcatfish]

Plan from @vdemeester in https://docs.google.com/document/d/1Tf3tdIBwkN5kxMuyYMo802F-DyOdsl_bgh-78a9prWQ/edit

[bobcatfish]

• These days dogfooding is in full swing and most issues are in https://github.com/tektoncd/plumbing
• This is the project we are using to track our current dogfooding efforts: https://github.com/orgs/tektoncd/projects/1
• This document contains our dogfooding plans