Prebuild issues roundup

[geropl]

Description

Hey @filiptronicek 👋

This PR was meant to address a lot of the current prebuild issues, but I did not mange to finish it last week. 😢
The key piece missing is that we render the log output 2 or 3 times now, so a react invalidation problem:

• I think the strategy of caching it by "prebuildId + taskId" in the respective tab's state can still work. But I was running out of time debugging where the invalidation&re-renders came from.
• If that turns out to be too hard (this was where some of the referenced bug came from: it would not invalidate e.g. on prebuildId change 👇 ), we can easily turn to caching in in localstorage - or even re-stream on every render and rely on HTTP caching - worked astonishingly good on the old PrebuildLogs.tsx)

Anyway, happy to sync for ~30mins a slot tomorrow morning if that helps. Also, happy to do the review! 🥳

Related Issue(s)

Meant to:

• fixes ENT-282 (fixes streaming bug in server by avoiding duplicating frontend logic in backend)
• fixes ENT-323 (shows SHA + duration)
• fixes ENT-368 (re-purpose un-unsed QUEUE staate for everything before RUNNING)
• fixes ENT-428 (we did miss to invalidate some components/the tabs when the prebuildId changed - now the keys are "prebuildId + taskId")

How to test

https://gpl-prebuild-cleanup.preview.gitpod-dev.com/workspaces

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

• 🏷️ Name - gpl-prebuild-cleanup
• 🔗 URL - gpl-prebuild-cleanup.preview.gitpod-dev.com/workspaces.
• 📚 Documentation - See our internal documentation for information on how to interact with your preview environment.
• 📦 Version - gpl-prebuild-cleanup-gha.27254
• 🗒️ Logs - GCP Logs Explorer

Build Options

Build

• /werft with-werft
  Run the build with werft instead of GHA
• leeway-no-cache
• /werft no-test
  Run Leeway with --dont-test

Publish

• /werft publish-to-npm
• /werft publish-to-jb-marketplace

Installer

• analytics=segment
• with-dedicated-emulation
• workspace-feature-flags
  Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• /werft preemptible
  Saves cost. Untick this only if you're really sure you need a non-preemtible machine.
• with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
• with-monitoring

/hold

[socket-security]

🚨 Potential security issues detected. Learn more about Socket for GitHub ↗︎

To accept the risk, merge this PR and you will not be notified again.

Alert	Package	Note	Source	CI
Install scripts	npm/grpc-tools@1.12.4	Install script: install Source: node-pre-gyp install	components/ide-metrics-api/typescript-grpc/package.json	🚫

View full report↗︎

Next steps

What is an install script?

Install scripts are run when the package is installed. The majority of malware in npm is hidden in install scripts.

Packages should not be running non-essential scripts during install and there are often solutions to problems people solve with install scripts that can be run at publish time instead.

Take a deeper look at the dependency

Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support [AT] socket [DOT] dev.

Remove the package

If you happen to install a dependency that Socket reports as Known Malware you should immediately remove it and select a different dependency. For other alert types, you may may wish to investigate alternative packages or consider if there are other ways to mitigate the specific risk posed by the dependency.

Mark a package as acceptable risk

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of ecosystem/package-name@version specifiers. e.g. @SocketSecurity ignore npm/foo@1.0.0 or ignore all packages with @SocketSecurity ignore-all

• @SocketSecurity ignore npm/grpc-tools@1.12.4

[mustard-mh]

Will review and test this PR later 🌮

[mustard-mh]

It's all working well in general, and we can do the rest in follow up.

1️⃣ If a prebuild has image-build, and image-build failed, prebuild detail page will stuck in Pending phase and no updates until user refreshes the page -> Turn into Error, message Error: headless task failed: exit status 1 and labels Duration and Stopped (time) shown

Example context https://github.com/mustard-mh/test/tree/prebuild/image-build-failed

Stuck in Pending	Refresh Page

2️⃣ nit: Click button to re-run prebuild, it will show Image build task tab then disappear (should be a follow up?)

3️⃣ Sometimes stopped (watch until it stopped) prebuild detail page doesn't show Stopped and Duration labels on top right until page refreshed I could not reproduce it anymore

4️⃣ nit: as we will show commit SHA on prebuild detail page, we could also show such label on prebuilds list page?

5️⃣ Before image build workspace created and recorded in db, dashboard will show inaccessible content for a long time, while as an user, I don't know if it's really running or pending or never able to access that logs

6️⃣ Re-run prebuild will use the latest commit instead of that commit SHA shows on prebuild detail page (SHA changed)

[filiptronicek]

@mustard-mh many thanks for the review!

1️⃣ If a prebuild has image-build, and image-build failed, prebuild detail page will stuck in Pending phase and no updates until user refreshes the page -> Turn into Error, message Error: headless task failed: exit status 1 and labels Duration and Stopped (time) shown

I created ENT-445 for this, nice find!

2️⃣ nit: Click button to re-run prebuild, it will show Image build task tab then disappear (should be a follow up?)

This boils down to us being unable to know whether an image build is going to happen from the shape we get from the API. Not sure if there's any quick wins to be had here.

3️⃣ Sometimes stopped (watch until it stopped) prebuild detail page doesn't show Stopped and Duration labels on top right until page refreshed I could not reproduce it anymore

Interestingly enough, I ran into this as well, but when I tried to show it to @geropl this morning, it didn't happen 😆. I think it means there is for sure an issue, but I could not pin it down yet.

Edit: ENT-450 exists now for this

4️⃣ nit: as we will show commit SHA on prebuild detail page, we could also show such label on prebuilds list page?

It used to be like this, but I removed it from the list because it took a lot of space in a table where we don't have much of it available.

5️⃣ Before image build workspace created and recorded in db, dashboard will show inaccessible content for a long time, while as an user, I don't know if it's really running or pending or never able to access that logs

This should most likely be dealt with separately, as this PR does not change this behavior. I agree we should make this part more reliable, but it might not be worth the effort.

6️⃣ Re-run prebuild will use the latest commit instead of that commit SHA shows on prebuild detail page (SHA changed)

I think this is fine as:

1. It was always like this
2. The ref which will be used is written on the button

[socket-security]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@types/google-protobuf@3.15.6	None	0	110 kB	types
npm/@types/node@18.0.4	None	0	1.73 MB	types
npm/cliui@8.0.1	None	+1	43 kB	oss-bot
npm/console-control-strings@1.1.0	None	0	12.7 kB	iarna
npm/google-protobuf@3.20.1	None	0	787 kB	haberman
npm/grpc_tools_node_protoc_ts@5.3.2	filesystem Transitive: environment, eval	+5	5.06 MB	agreatfool
npm/grpc-tools@1.12.4	Transitive: environment, filesystem, network, shell	+48	1.82 MB	murgatroid99
npm/inherits@2.0.4	None	0	3.96 kB	isaacs
npm/long@5.2.3	None	0	119 kB	dcode
npm/protobufjs@7.3.2	filesystem, network	+9	2.97 MB	dcode
npm/string-width@4.2.3	None	+2	58.4 kB	sindresorhus
npm/strip-ansi@6.0.1	None	+1	9.64 kB	sindresorhus
npm/typescript-formatter@7.2.2	filesystem Transitive: environment	+7	414 kB	vvakame
npm/typescript@4.4.4	None	0	62.7 MB	typescript-bot
npm/yargs-parser@21.1.1	environment, filesystem	0	128 kB	oss-bot
npm/yargs@17.7.2	environment, filesystem	+4	344 kB	oss-bot

🚮 Removed packages: npm/aria-hidden@1.2.3, npm/array-includes@3.1.6, npm/arrify@2.0.1, npm/asap@2.0.6, npm/asn1.js@5.4.1, npm/assertion-error@1.1.0, npm/async-batch@1.1.2, npm/autoprefixer@10.4.15, npm/axios@0.21.4, npm/babel-jest@27.5.1, npm/base-64@1.0.0, npm/base64-js@1.5.1, npm/base64url@3.0.1, npm/before-after-hook@2.2.2, npm/bignumber.js@9.1.2, npm/bitbucket@2.7.0, npm/bluebird@3.7.2, npm/bn.js@4.12.0, npm/body-parser@1.19.2, npm/boolbase@1.0.0, npm/braces@3.0.2, npm/brorand@1.1.0, npm/browser-stdout@1.3.1, npm/browserify-aes@1.2.0, npm/browserify-cipher@1.0.1, npm/browserify-rsa@4.1.0, npm/browserify-sign@4.2.1, npm/browserslist@4.21.10, npm/buffer@4.9.2, npm/bufrw@1.3.0, npm/builtins@5.0.1

View full report↗︎