Add ability for project clone to 'bootstrap' devworkspaces #1193
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add attribute `controller.devfile.io/bootstrap-workspace`, which
configures the project-clone container to
1. Clone all projects in the DevWorkspace as normal
2. Search cloned projects for a devfile.yaml or .devfile.yaml
3. Merge contents of this devfile into the DevWorkspace
* Projects are merged (i.e. both devfile and DevWorkspace projects
are used)
* Attributes from the DevWorkspace override attributes from the
devfile
* Components, commands, events, etc. from the Devfile are used.
4. Apply the updated DevWorkspace to the cluster, automatically
restarting the workspace.
This allows starting workspaces from repositories that contain a devfile
without first loading the repository to search for this devfile by
creating a basic workspace and enabling project clone to bootstrap it.
Signed-off-by: Angel Misevski <amisevsk@redhat.com>
Signed-off-by: Angel Misevski <amisevsk@redhat.com>
|
@amisevsk looks awesome to me so far 🥳 Will aim to do a full review on Monday. |
This was referenced Oct 20, 2023
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1193 +/- ##
==========================================
- Coverage 52.96% 52.81% -0.15%
==========================================
Files 84 84
Lines 7583 7592 +9
==========================================
- Hits 4016 4010 -6
- Misses 3278 3292 +14
- Partials 289 290 +1
☔ View full report in Codecov by Sentry. |
AObuchow
approved these changes
Oct 24, 2023
There was a problem hiding this comment.
Code looks good to me & works as expected.
I tested this with Che using the following devfile:
schemaVersion: 2.1.0
metadata:
name: java-backend
projects:
- name: che-dashboard
git:
remotes:
origin: "https://github.com/eclipse-che/che-dashboard.git"
attributes:
controller.devfile.io/bootstrap-devworkspace: true
components:
- name: tools
container:
image: quay.io/devfile/universal-developer-image:ubi8-latest
memoryLimit: 3Gi The resulting devworkspace after the bootstrap process completed (see below) resembled the devworkspace made from the che-dashboard devfile:
apiVersion: workspace.devfile.io/v1alpha2
kind: DevWorkspace
metadata:
annotations:
che.eclipse.org/devfile: |
schemaVersion: 2.1.0
metadata:
name: java-backend
namespace: che-kube-admin-che-8ppidy
projects:
- name: che-dashboard
git:
remotes:
origin: https://github.com/eclipse-che/che-dashboard.git
attributes:
controller.devfile.io/bootstrap-devworkspace: true
dw.metadata.annotations:
che.eclipse.org/devfile-source: |
scm:
repo: https://github.com/AObuchow/java-devfile-test.git
revision: main
fileName: devfile.yaml
factory:
params: url=https://github.com/AObuchow/java-devfile-test/tree/main
components:
- name: tools
container:
image: quay.io/devfile/universal-developer-image:ubi8-latest
memoryLimit: 3Gi
che.eclipse.org/last-updated-timestamp: '2023-10-24T18:45:30.595Z'
controller.devfile.io/started-at: '1698173168348'
creationTimestamp: '2023-10-24T18:45:29Z'
finalizers:
- storage.controller.devfile.io
- rbac.controller.devfile.io
generation: 6
labels:
controller.devfile.io/creator: ''
managedFields:
(...)
manager: devworkspace-controller
operation: Update
subresource: status
time: '2023-10-24T18:46:08Z'
name: java-backend
namespace: che-kube-admin-che-8ppidy
resourceVersion: '83524'
uid: 2fb0c4cc-1dab-478c-9735-1b9f8076e35a
spec:
contributions:
- kubernetes:
name: che-code-java-backend
name: editor
routingClass: che
started: true
template:
attributes:
controller.devfile.io/devworkspace-config:
name: devworkspace-config
namespace: openshift-operators
controller.devfile.io/scc: container-build
controller.devfile.io/storage-type: per-user
dw.metadata.annotations:
che.eclipse.org/devfile-source: |
scm:
repo: https://github.com/AObuchow/java-devfile-test.git
revision: main
fileName: devfile.yaml
factory:
params: url=https://github.com/AObuchow/java-devfile-test/tree/main
commands:
- exec:
commandLine: yarn install --non-interactive
component: tools
group:
isDefault: true
kind: build
label: '[UD] install dependencies'
workingDir: ${PROJECT_SOURCE}
id: installdependencies
- exec:
commandLine: yarn install --non-interactive && yarn build
component: tools
group:
kind: build
label: '[UD] build'
workingDir: ${PROJECT_SOURCE}
id: build
- exec:
commandLine: yarn --cwd packages/dashboard-frontend build:watch
component: tools
group:
kind: build
label: '[UD] watch frontend'
workingDir: ${PROJECT_SOURCE}
id: watchfrontend
- exec:
commandLine: yarn --cwd packages/dashboard-backend build:watch
component: tools
group:
kind: build
label: '[UD] watch backend'
workingDir: ${PROJECT_SOURCE}
id: watchbackend
- exec:
commandLine: yarn start:prepare
component: tools
group:
kind: build
label: '[UD] prepare'
workingDir: ${PROJECT_SOURCE}
id: prepare
- exec:
commandLine: >-
oc project dogfooding && export CHE_NAMESPACE='dogfooding' && export
NAMESPACE='dogfooding' && yarn start:prepare && yarn start -c
component: tools
group:
kind: build
label: '[UD] dogfooding start'
workingDir: ${PROJECT_SOURCE}
id: dogfoodingprepare
- exec:
commandLine: yarn start:prepare && yarn start -c
component: tools
group:
kind: build
label: '[UD] start che-in-che'
workingDir: ${PROJECT_SOURCE}
id: startcheinche
- exec:
commandLine: yarn test
component: tools
group:
kind: test
label: '[UD] test'
workingDir: ${PROJECT_SOURCE}
id: runtests
- exec:
commandLine: >-
yarn --cwd packages/dashboard-frontend build:dev --env
speedMeasure=true
component: tools
group:
kind: build
label: '[UD] frontend build speed'
workingDir: ${PROJECT_SOURCE}
id: frontendbuildspeed
- exec:
commandLine: >-
yarn --cwd packages/dashboard-backend build:dev --env
speedMeasure=true
component: tools
group:
kind: build
label: '[UD] backend build speed'
workingDir: ${PROJECT_SOURCE}
id: backendbuildspeed
- exec:
commandLine: >-
yarn --cwd packages/dashboard-frontend build --env
bundleAnalyzer=true
component: tools
group:
kind: build
label: '[UD] frontend bundle analyzer'
workingDir: ${PROJECT_SOURCE}
id: frontendbundleanalyzer
- exec:
commandLine: >-
yarn --cwd packages/dashboard-backend build --env
bundleAnalyzer=true
component: tools
group:
kind: build
label: '[UD] backend bundle analyzer'
workingDir: ${PROJECT_SOURCE}
id: backendbundleanalyzer
components:
- container:
cpuLimit: 5000m
cpuRequest: 1000m
endpoints:
- exposure: public
name: local-server
path: /
protocol: http
targetPort: 8080
- exposure: public
name: bundle-analyzer
path: /
protocol: http
targetPort: 8888
env:
- name: KUBEDOCK_ENABLED
value: 'true'
image: quay.io/okurinny/dashboard.dev:skaffold
memoryLimit: 10G
memoryRequest: 512Mi
mountSources: true
sourceMapping: /projects
name: tools
projects:
- git:
remotes:
origin: https://github.com/eclipse-che/che-dashboard.git
name: che-dashboard
status:
conditions:
- lastTransitionTime: '2023-10-24T18:45:30Z'
message: DevWorkspace is starting
status: 'True'
type: Started
- lastTransitionTime: '2023-10-24T18:45:30Z'
message: Resolved plugins and parents from DevWorkspace
status: 'True'
type: DevWorkspaceResolved
- lastTransitionTime: '2023-10-24T18:45:30Z'
message: Storage ready
status: 'True'
type: StorageReady
- lastTransitionTime: '2023-10-24T18:45:51Z'
message: Networking ready
status: 'True'
type: RoutingReady
- lastTransitionTime: '2023-10-24T18:45:51Z'
message: DevWorkspace serviceaccount ready
status: 'True'
type: ServiceAccountReady
- lastTransitionTime: '2023-10-24T18:45:51Z'
message: DevWorkspace secrets ready
status: 'True'
type: PullSecretsReady
- lastTransitionTime: '2023-10-24T18:46:07Z'
message: DevWorkspace deployment ready
status: 'True'
type: DeploymentReady
- lastTransitionTime: '2023-10-24T18:46:08Z'
status: 'True'
type: Ready
devworkspaceId: workspace2fb0c4cc1dab478c
mainUrl: >-
https://eclipse-che.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com/kube-admin/java-backend/3100/
message: >-
https://eclipse-che.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com/kube-admin/java-backend/3100/
phase: RunningFor reference, here's the devworkspace resulting from the Che Dashboard devfile:
apiVersion: workspace.devfile.io/v1alpha2
kind: DevWorkspace
metadata:
annotations:
che.eclipse.org/devfile: |
schemaVersion: 2.1.0
metadata:
name: che-dashboard
namespace: che-kube-admin-che-8ppidy
components:
- name: tools
container:
image: quay.io/okurinny/dashboard.dev:skaffold
memoryLimit: 10G
memoryRequest: 512Mi
cpuRequest: 1000m
cpuLimit: 5000m
mountSources: true
endpoints:
- exposure: public
name: local-server
protocol: http
targetPort: 8080
path: /
- exposure: public
name: bundle-analyzer
path: /
protocol: http
targetPort: 8888
env:
- name: KUBEDOCK_ENABLED
value: 'true'
commands:
- id: installdependencies
exec:
label: '[UD] install dependencies'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn install --non-interactive
group:
kind: build
isDefault: true
- id: build
exec:
label: '[UD] build'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn install --non-interactive && yarn build
group:
kind: build
- id: watchfrontend
exec:
label: '[UD] watch frontend'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn --cwd packages/dashboard-frontend build:watch
group:
kind: build
- id: watchbackend
exec:
label: '[UD] watch backend'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn --cwd packages/dashboard-backend build:watch
group:
kind: build
- id: prepare
exec:
label: '[UD] prepare'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn start:prepare
group:
kind: build
- id: dogfoodingprepare
exec:
label: '[UD] dogfooding start'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: >-
oc project dogfooding && export CHE_NAMESPACE='dogfooding' && export
NAMESPACE='dogfooding' && yarn start:prepare && yarn start -c
group:
kind: build
- id: startcheinche
exec:
label: '[UD] start che-in-che'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn start:prepare && yarn start -c
group:
kind: build
- id: runtests
exec:
label: '[UD] test'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn test
group:
kind: test
- id: frontendbuildspeed
exec:
label: '[UD] frontend build speed'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: >-
yarn --cwd packages/dashboard-frontend build:dev --env
speedMeasure=true
group:
kind: build
- id: backendbuildspeed
exec:
label: '[UD] backend build speed'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn --cwd packages/dashboard-backend build:dev --env speedMeasure=true
group:
kind: build
- id: frontendbundleanalyzer
exec:
label: '[UD] frontend bundle analyzer'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn --cwd packages/dashboard-frontend build --env bundleAnalyzer=true
group:
kind: build
- id: backendbundleanalyzer
exec:
label: '[UD] backend bundle analyzer'
component: tools
workingDir: ${PROJECT_SOURCE}
commandLine: yarn --cwd packages/dashboard-backend build --env bundleAnalyzer=true
group:
kind: build
projects:
- name: che-dashboard
git:
remotes:
origin: https://github.com/eclipse-che/che-dashboard.git
attributes:
dw.metadata.annotations:
che.eclipse.org/devfile-source: |
scm:
repo: https://github.com/eclipse-che/che-dashboard.git
fileName: devfile.yaml
factory:
params: url=https://github.com/eclipse-che/che-dashboard
che.eclipse.org/last-updated-timestamp: '2023-10-24T18:14:35.133Z'
controller.devfile.io/started-at: '1698171427903'
creationTimestamp: '2023-10-24T18:14:34Z'
finalizers:
- storage.controller.devfile.io
- rbac.controller.devfile.io
generation: 5
labels:
controller.devfile.io/creator: ''
managedFields:
(...)
manager: devworkspace-controller
operation: Update
subresource: status
time: '2023-10-24T18:17:07Z'
name: che-dashboard
namespace: che-kube-admin-che-8ppidy
resourceVersion: '70085'
uid: 35cdb00b-90bb-48ae-a15f-354d92e3f26d
spec:
contributions:
- kubernetes:
name: che-code-che-dashboard
name: editor
routingClass: che
started: true
template:
attributes:
controller.devfile.io/devworkspace-config:
name: devworkspace-config
namespace: openshift-operators
controller.devfile.io/scc: container-build
controller.devfile.io/storage-type: per-user
dw.metadata.annotations:
che.eclipse.org/devfile-source: |
scm:
repo: https://github.com/eclipse-che/che-dashboard.git
fileName: devfile.yaml
factory:
params: url=https://github.com/eclipse-che/che-dashboard
commands:
- exec:
commandLine: yarn install --non-interactive
component: tools
group:
isDefault: true
kind: build
label: '[UD] install dependencies'
workingDir: '${PROJECT_SOURCE}'
id: installdependencies
- exec:
commandLine: yarn install --non-interactive && yarn build
component: tools
group:
kind: build
label: '[UD] build'
workingDir: '${PROJECT_SOURCE}'
id: build
- exec:
commandLine: 'yarn --cwd packages/dashboard-frontend build:watch'
component: tools
group:
kind: build
label: '[UD] watch frontend'
workingDir: '${PROJECT_SOURCE}'
id: watchfrontend
- exec:
commandLine: 'yarn --cwd packages/dashboard-backend build:watch'
component: tools
group:
kind: build
label: '[UD] watch backend'
workingDir: '${PROJECT_SOURCE}'
id: watchbackend
- exec:
commandLine: 'yarn start:prepare'
component: tools
group:
kind: build
label: '[UD] prepare'
workingDir: '${PROJECT_SOURCE}'
id: prepare
- exec:
commandLine: >-
oc project dogfooding && export CHE_NAMESPACE='dogfooding' && export
NAMESPACE='dogfooding' && yarn start:prepare && yarn start -c
component: tools
group:
kind: build
label: '[UD] dogfooding start'
workingDir: '${PROJECT_SOURCE}'
id: dogfoodingprepare
- exec:
commandLine: 'yarn start:prepare && yarn start -c'
component: tools
group:
kind: build
label: '[UD] start che-in-che'
workingDir: '${PROJECT_SOURCE}'
id: startcheinche
- exec:
commandLine: yarn test
component: tools
group:
kind: test
label: '[UD] test'
workingDir: '${PROJECT_SOURCE}'
id: runtests
- exec:
commandLine: >-
yarn --cwd packages/dashboard-frontend build:dev --env
speedMeasure=true
component: tools
group:
kind: build
label: '[UD] frontend build speed'
workingDir: '${PROJECT_SOURCE}'
id: frontendbuildspeed
- exec:
commandLine: >-
yarn --cwd packages/dashboard-backend build:dev --env
speedMeasure=true
component: tools
group:
kind: build
label: '[UD] backend build speed'
workingDir: '${PROJECT_SOURCE}'
id: backendbuildspeed
- exec:
commandLine: >-
yarn --cwd packages/dashboard-frontend build --env
bundleAnalyzer=true
component: tools
group:
kind: build
label: '[UD] frontend bundle analyzer'
workingDir: '${PROJECT_SOURCE}'
id: frontendbundleanalyzer
- exec:
commandLine: >-
yarn --cwd packages/dashboard-backend build --env
bundleAnalyzer=true
component: tools
group:
kind: build
label: '[UD] backend bundle analyzer'
workingDir: '${PROJECT_SOURCE}'
id: backendbundleanalyzer
components:
- container:
cpuLimit: 5000m
cpuRequest: 1000m
endpoints:
- exposure: public
name: local-server
path: /
protocol: http
targetPort: 8080
- exposure: public
name: bundle-analyzer
path: /
protocol: http
targetPort: 8888
env:
- name: KUBEDOCK_ENABLED
value: 'true'
- name: CHE_DASHBOARD_URL
value: >-
https://eclipse-che.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com
- name: CHE_PLUGIN_REGISTRY_URL
value: >-
https://eclipse-che.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com/plugin-registry/v3
- name: CHE_PLUGIN_REGISTRY_INTERNAL_URL
value: 'http://plugin-registry.openshift-operators.svc:8080/v3'
- name: CLUSTER_CONSOLE_URL
value: >-
https://console-openshift-console.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com
- name: CLUSTER_CONSOLE_TITLE
value: OpenShift console
- name: OPENVSX_REGISTRY_URL
value: 'https://open-vsx.org'
image: 'quay.io/okurinny/dashboard.dev:skaffold'
memoryLimit: 10G
memoryRequest: 512Mi
mountSources: true
sourceMapping: /projects
name: tools
projects:
- git:
remotes:
origin: 'https://github.com/eclipse-che/che-dashboard.git'
name: che-dashboard
status:
conditions:
- lastTransitionTime: '2023-10-24T18:14:35Z'
message: DevWorkspace is starting
status: 'True'
type: Started
- lastTransitionTime: '2023-10-24T18:14:35Z'
message: Resolved plugins and parents from DevWorkspace
status: 'True'
type: DevWorkspaceResolved
- lastTransitionTime: '2023-10-24T18:14:35Z'
message: Storage ready
status: 'True'
type: StorageReady
- lastTransitionTime: '2023-10-24T18:14:36Z'
message: Networking ready
status: 'True'
type: RoutingReady
- lastTransitionTime: '2023-10-24T18:14:36Z'
message: DevWorkspace serviceaccount ready
status: 'True'
type: ServiceAccountReady
- lastTransitionTime: '2023-10-24T18:14:36Z'
message: DevWorkspace secrets ready
status: 'True'
type: PullSecretsReady
- lastTransitionTime: '2023-10-24T18:17:07Z'
message: DevWorkspace deployment ready
status: 'True'
type: DeploymentReady
- lastTransitionTime: '2023-10-24T18:17:07Z'
status: 'True'
type: Ready
devworkspaceId: workspace35cdb00b90bb48ae
mainUrl: >-
https://eclipse-che.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com/kube-admin/che-dashboard/3100/
message: >-
https://eclipse-che.apps.ci-ln-5689ss2-72292.origin-ci-int-gce.dev.rhcloud.com/kube-admin/che-dashboard/3100/
phase: RunningI also was able to verify that running devfile commands as tasks worked as expected in Che Code.
|
@svor @RomanNikitenko If you have some time, please also take a look at this PR and leave review comments (I can't explicitly request review from you). This is somewhat related to the devworkspace generator and the "Restart from local Devfile" functionality in Code (e.g. this PR). This PR partly reimplements the logic/goal of the generator's
If you would like me to contribute something similar to the generator, let me know :) |
|
@amisevsk |
I don't think the generator or che-code needs to be updated, and there's not really an issue on that side of things:
On the other hand, the DevWorkspace/devfile/Che APIs have reached a level of complexity that it's probably worth our time to discuss and make sure our interpretations of how things should fit together are in line -- I don't want to do something in DWO that is unexpected from the perspective of the devworkspace-generator library. As for projects, the implementation here is basically just a temporary workaround until dependentProjects are added to the devfile API. After dependentProjects is introduced, handling this is a fair bit simpler :) |
svor
approved these changes
Oct 30, 2023
There was a problem hiding this comment.
LGTM
I don't think any changes are required on the generator side.
For me it looks like one more way to apply project's devfile having a short predefined devWorkspace and it applies only default editor, since generator tool could be configured (via editor-entry parameter) to apply any editor.
I'm curious where this flow could be useful/applicable in case of Che/DS? Maybe in tests that don't use UI and were we need to have a final DevWorkspace, but we still need to create and store short version of DevWorkspace with bootstrap-devworkspace attribute.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: amisevsk, AObuchow, svor The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
I don't have anything specific in mind, it was just brought up that this change may be of interest to the generator (and prompted me to look into the generator as well to see how it works). I agree that the generator is doing something different (and needs to generate e.g. devworkspacetemplates, etc.). One area we could potentially improve is eclipse-che/che#22614 -- currently the generator will set environment variables on all containers when converting the devfile, and I'm not sure that's still necessary with the move to container contributions (which merge environment variables from the contribution). As far as I know, those Che-specific environment variables are needed only by the editor, so we could probably get away with not setting them in any devfile containers. This would also make bootstrapping "safer" since the result of bootstrapping would be more similar to the result of the generator. |
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds DevWorkspace attribute
controller.devfile.io/bootstrap-devworkspace. If set totrue, this attribute will configure project clone to do the following:are used)
devfile
restarting the workspace.
What issues does this PR fix or reference?
Closes #1192
Is it tested? How?
Basic testing of this PR is to create a DevWorkspace that uses the attribute and imports a project with a devfile.yaml at its root:
After starting, the workspace should update so that the
.spec.templateof the workspace matches the devfile of the imported project, with the merge process above (attributes, projects are merged instead of overwritten)PR Checklist
/test v8-devworkspace-operator-e2e, v8-che-happy-pathto trigger)v8-devworkspace-operator-e2e: DevWorkspace e2e testv8-che-happy-path: Happy path for verification integration with Che