Ensure release artifacts are passing some basic tests like Happy Path tests before pushing chectl release binaries

[benoitf]

Is your enhancement related to a problem? Please describe.

che-plugin-registry released image had invalid data in it. But all 'release process' was successful as jobs being successfully terminated.
But the resulting latest release was failing.
We should prevent to make publicly available a release if it's not passing some basic tests.

Describe the solution you'd like

Before pushing chectl release artifacts, ensure that the tagged Che version is passing Happy Path tests.
If it does not pass, we could hold the chectl release and then end-users don't see a new version available.

Describe alternatives you've considered

Of course, each component could have its own checks as post-release check but a end-to-end check would ensure that something basic is still working.

Additional context

[tolusha]

@benoitf
I think it is too later to check before pushing chectl release artifacts.
The verification steps must be done at che-plugin-registry level

[benoitf]

@tolusha
Well there is #18747 for adding more checks to che-plugin-registry as well but it does not mean that if all individual checks are ok that the release is still OK.

So we need to test what is being made before making it 'public' to users.

[sunix]

I am setting that one to blocker: what ever the origin of the problem, we can not just release something that is broken.

[sunix]

at the moment 7.23.2 is just not working. So anyone installing che at the moment has a broken product ...

[tolusha]

What exactly doesn't work ?
There are basic tests during releasing che-operator on minikube, minishift, ocp 4.x platforms:

1. Deploying Eclipse Che previous stable version
2. Start a workspace from an empty devfile
3. Update Eclipse Che to a release version
4. Start a workspace

[sunix]

try to start a workspace with the typescript plugin for instance ... it is failling to start.

[sunix]

same for the quarkus plugin

Java

[sunix]

So anyone who install or upgrade Che to the stable version (7.23.2 atm) can't start any workspace with plugins ... that is pretty blocker to me. It may be fixed for 7.24.0 but I think it's critical that we have a minimum of integration test for a release

[tolusha]

I see. We will improve our tests on che-operator side.

[sunix]

It is even not straight foward how to use another registry ...

[sunix]

I am even not sure why it is trying to reach prod preview registry :/ (I pointed the registry to another location)

[dmytro-ndp]

It's possible to run Happy path tests and E2E Devfile tests against already released version of Eclipse Che even now by building the next Jenkins job with proper set of parameters, if chectl binary of released version is existed https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/job/Che/job/e2e/job/minikube/job/basic/job/eclipse-che/

[nickboldt]

So that there's an owner here for the Automated Backlog, I've set Dmytro. If that's an error, please set it to someone else.

[tolusha]

Happy Path tests added to chectl.
Can we close this issue?

[benoitf]

@tolusha yes