Create acceptance tests to go through all devfile samples and all editors using API

[dmytro-ndp]

Is your task related to a problem? Please describe

There are tens of devfile samples and several supported editors which are not covered by automated acceptance tests.

Describe the solution you'd like

Samples in upstream and with supported editors have to be covered by automated tests:

• open workspaces in certain editor;
• execute each available command described in devfile from within workspace.

The goal is to run devfile API tests instead of GUI tests with complex scenario, a.k.a Happy Path tests.

Describe alternatives you've considered

No response

Additional context

No response

[nallikaea]

@dmytro-ndp @l0rd @musienko-maxim colleagues, could you, please, update this description?

[dmytro-ndp]

@nallikaea : done.

[l0rd]

• Should be used as a PR check for UDI
• The list of samples should be grabbed automatically from the devfile registry main branch
• The PR Check should be reusable for other repos such as the che devfile registry and cncf devfile registry
• The core test should be:

  1. Take in input a git repository URL or a devfile raw URL
  2. Transform the devfile in a DevWorkspace + VS Code DevWorkspaceTemplate
  3. Apply the DevWorkspace and wait until it's started
  4. Verify that the project has been cloned successfully
  5. Execute build commands and verify that they complete successfully
  6. Execute run/debug commands and verify the lifeness probe
  

• Build commands are commands with group.kind: build:

  commands:
    - id: build-project
      exec:
        commandLine: JAVA_HOME=$JAVA_HOME_8 mvn clean install
        component: tools
        workingDir: "${PROJECT_SOURCE}"
        group:
          kind: build

• Run/debug commands are commands that and require a livenessProbe attribute such as:

    commands:
      - id: run-services
        attributes:
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              httpHeaders:
              - name: Custom-Header
                value: Awesome
            initialDelaySeconds: 3
            periodSeconds: 3
        exec:
          commandLine: JAVA_HOME=$JAVA_HOME_8 mvn spring-boot:run -DskipTests
          component: tools
          workingDir: "${PROJECT_SOURCE}"

• Commands that do not have group.kind: build and do not have attributs.livenessProbe should be skipped

[nallikaea]

Subtask: #22212

[nallikaea]

Subtask: #22229

[nallikaea]

5. Execute build commands and verify that they complete successfully

• Build commands are commands with group.kind: build:

  commands:
    - id: build-project
      exec:
        commandLine: JAVA_HOME=$JAVA_HOME_8 mvn clean install
        component: tools
        workingDir: "${PROJECT_SOURCE}"
        group:
          kind: build

• Commands that do not have group.kind: build and do not have attributs.livenessProbe should be skipped

@l0rd in Apache Camel based on Spring Boot sample devfile we have build command without properties group: kind: build. Should we update it?

[l0rd]

Should we update it?

Yes, I have created a PR

[ibuziuk]

Create acceptance tests to go through all devfile samples and all editors using API

whereas testing all editors using API makes sense, not sure if we should cover all the devfiles from the internal registry given that in the short run, we are going to switch to the devfile.io. IMO, it would be progmatic to have the skeleton with an empty workspace test for the time being, and start adding to it devfiles from devfile.io e.g. UDI - https://registry.devfile.io/viewer/devfiles/community/udi

@l0rd wdyt?

[musienko-maxim]

So, I brought up the test code to the actual state. Locally, the test ran successfully. But on the GitHub infra, I got problems with the limitations of VM, which is provided on the GitHub side. I think we can close this issue and make decisions for alternative solutions:
The example of the job here: https://github.com/devfile/developer-images/actions/runs/6981551741/job/18998924174?pr=114
Potential solutions:

• Extend the GitHub workflow instance
• Run tests on alternative infrastructures (OpenShift CI, PSI, other)