Ability to run tests of a vscode extension on top of che-theia

[benoitf]

Is your task related to a problem? Please describe.

A VS Code extension can declare some tests that can be run usually with test script.
https://code.visualstudio.com/api/working-with-extensions/testing-extension

It would be great to be able to execute these tests on top of a Che-Theia instance to validate if the extension behaves as expected.

Describe the solution you'd like

Have a way to easily provide an extension (or a set of extensions) and grab the result of the tests.

Additional context

Some of the vscode-test module is unpacking a VS Code instance, etc so we need to be sure that it can be launched as expected on top of Che-Theia

[svor]

@benoitf As you wrote in the description some extensions like helloworld-test-sample download VS Code and run the Mocha tests on it. I've made it possible to run these tests from che in VS Code instance (i've just prepared an image for that) but I don't see the reason of it.
So, am I correct that we have to override this operation to run these tests in che instance NOT in VS Code.

[benoitf]

@svor yes it needs to tests within a Che instance, not VS Code (the goal is to check if tests are passing in Che, not on VS Code)

[svor]

@benoitf @ericwill
Could you please take a look at this instructions which I'm trying to implement right now and please let me know if you see some concerns or problems here.

The main idea is:

1. Start a workspace by using the devfile which includes:

   • A sidecar plugin with VSCode extension which should be tested
   • Tested extension's source code (clone into /projects)
   • New theia plugin which will collect all tests from VSCode extension, run them and build the report.

So, the devfile will be like:

apiVersion: 1.0.0
metadata:
  name: test-runner
projects:
  - name: vscode-yaml //extension which needs to be tested
    source:
      location: 'https://github.com/redhat-developer/vscode-yaml.git'
      type: git
      branch: master
  - name: vscode-extension-test-runner
    source:
      location: 'https://github.com/vscode-extension-test-runner.git'
      type: git
      branch: main
components:
  - type: chePlugin
    reference: 'https://raw.githubusercontent.com/vscode-extension-test-runner/meta.yaml'
  - id: redhat/vscode-yaml/latest
    type: chePlugin
  - id: eclipse/che-theia/next
    memoryLimit: 1024Mi
    type: cheEditor

2. When the workspace was started, the next steps are:
   • Build VS Code extension's source code
   • Execute some registered command by the new theia plugin to collect Mocha tests and run them
   • Generate testReport file with the results

[ericwill]

I think it's a good first step. The end goal is to automate this in a CI job, whenever a new plugin is added/updated.

1. When the workspace was started, the next steps are:
   
   * Build VS Code extension's source code
   * Execute some registered command by the new theia plugin to collect Mocha tests and run them
   * Generate testReport file with the results

I guess the tests have to be run at build time, correct?

[benoitf]

@svor I would say that intermediate devfile is not so important on the goal. (detail of implementation that belongs to you)

as Eric was saying:
When I submit a change to che-theia-plugins.yaml for every new VS Code extension added or updated, I need to run the tests contained by the provided Vsix files.

so yes basically it should contain a minkube deployment, che deployment and run a custom workspace based on a devfile

When the workspace was started, the next steps are:
Build VS Code extension's source code
Execute some registered command by the new theia plugin to collect Mocha tests and run them
Generate testReport file with the results

Here it's what is uncleared to me. You may just add different parameters to a Che-Theia instance starting in "test mode" and will execute tests and exit at the end. (0 = succeed, 1 = failure) as it's done on VS Code tests.

[ericwill]

@svor I would say that intermediate devfile is not so important on the goal. (detail of implementation that belongs to you)

I would say we could include such a devfile in the plugin registry repo, as an example for users who want to test an extension inside their Che.

[benoitf]

maybe if you want to test from inside Che, you might just want to run a command

open che-theia terminal

$ che-theia test /path-to-my.vsix

or if you have source code of your extension, you could do right click / test my extension in Che

but yes I'm not against having examples

[svor]

When I submit a change to che-theia-plugins.yaml for every new VS Code extension added or updated, I need to run the tests contained by the provided Vsix files.

As I know vsix files don't contain tests

[benoitf]

I mean' if you change in che-theia-plugins file some extensions/vsix then we need to test these VCode extensions. And we've repository/revision to grab associated source code ( of at least one extension) or if test folder is embedded in vsix

[svor]

The current status

I've created an utility Theia pluin che-vscode-extension-tests. This plug-in provides a command which collects all *.test.js files from /projects folder and runs them one by one by using mocha. Also this plug-in contains a devfile which was adapted to work with helloworld-test-sample - it is a simple VS Code extension sample which shows how to write tests.

Devfile

  apiVersion: 1.0.0
metadata:
  name: che-helloworld-test-sample
projects:
  - name: helloworld-test-sample
    source:
      location: 'https://github.com/svor/helloworld-test-sample.git'
      type: git
      branch: main
components:
  - mountSources: true
    memoryLimit: 3Gi
    type: dockerimage
    image: 'quay.io/eclipse/che-theia-dev:next'
    alias: che-dev
  - type: chePlugin
    reference: 'https://raw.githubusercontent.com/svor/che-vscode-extension-tests/main/meta.yaml'
  - id: eclipse/che-theia/next
    memoryLimit: 1024Mi
    type: cheEditor
commands:
  - name: 1. Build extension
    actions:
    - type: exec
      component: che-dev
      command: yarn && yarn compile
      workdir: ${CHE_PROJECTS_ROOT}/helloworld-test-sample

Che plug-in meta.yaml

apiVersion: v2
publisher: Red Hat
name: Smoke-Test-Plugin
version: 0.0.1
type: Theia plugin
displayName: Smoke Test Plugin
title: Smoke-Test-Plugin
description: This plug-in runs tests
icon: https://www.eclipse.org/che/images/logo-eclipseche.svg
repository: https://github.com/svor/che-vscode-extension-tests
category: Language
firstPublicationDate: "2021-01-28"
spec:
  extensions:
    - https://raw.githubusercontent.com/svor/che-vscode-extension-tests/main/che_vscode_test_theia_plugin.theia
    - https://raw.githubusercontent.com/svor/helloworld-test-sample/main/helloworld-sample-0.0.1.vsix

If start the a workspace by using this devfile, it will initialize 2 plugins: the one is VS Extension which needs to be tested and another to collect and execute tests, also it'll clone source code of the tested extension into the workspace. By executing 1.Build extension command we build the VS Code extension's source code to prepare tests files.

When /projects/tested_extension/**/*test.js files were generated we can run them by using the command from utility Theia plu-gin:

The result of tests execution was entered into /projects/test.log file.

TODO

The next my step is to automate this flow and run tests when some of the vsix url from che-theia-plugins.yaml will be changed. For this I need:

• Analyze somehow which extension is going to be updated and get source's url
• Generate a new devfile with a project from the task1, with a tested plug-in and utility plug-in to collect and run tests
• In the workspace build the extension's source code and run tests (make it automatically).

[svor]

@ericwill looks like the decision to grep tests from the extension which was cloned and built in /projects folder was wrong. Some execution of vscode API doesn't work as expected, for example vscode.extensions always returns undefined(that's why I've created an issue for that #19172), but for example vscode.workspace.* works as expected.
I've created a VSCode extension to check tests execution

apiVersion: 1.0.0
metadata:
  name: che-helloworld-test-sample
projects:
  - name: extension-tests-runner
    source:
      location: 'https://github.com/svor/extension-tests-runner.git'
      type: git
components:
  - mountSources: true
    memoryLimit: 3Gi
    type: dockerimage
    image: 'quay.io/eclipse/che-theia-dev:next'
    alias: che-dev
  - type: chePlugin
    reference: 'https://raw.githubusercontent.com/svor/extension-tests-runner/main/meta.yaml'
  - id: timonwong/shellcheck/latest
    type: chePlugin
  - id: eclipse/che-theia/next
    memoryLimit: 1024Mi
    type: cheEditor

And tried to execute the same tests from different places:

1. From /projects/extension-tests-runner/out/test/suit/extension.test.js (Run VSCode Extension's Tests From /projects command)
2. From the extension which is running in theia-ide container /tmp/vscode-unpacked/extension-tests-runner/out/test/suit/extension.test.js (Run VSCode Extension's Tests from extension command)

In the first case it was failed but in the second it was successfully finished.
So, needs to find another way how to execute extension's tests

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[svor]

/remove-lifecycle stale

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.