Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VSCode npm support #384

Draft
wants to merge 70 commits into
base: master
Choose a base branch
from
Draft

Conversation

Kawanaao
Copy link
Contributor

@Kawanaao Kawanaao commented Oct 18, 2024

Adaptation of the VSCode ebuild for npm, which VSCode switched to in newer versions. See issue #377

  • npm cache for offline (tar) building
  • VSCode builds in offline mode
  • VSCode builds in online mode
  • VSCode works as intended

@Kawanaao Kawanaao marked this pull request as draft October 18, 2024 17:23
@Kawanaao
Copy link
Contributor Author

It seems that the workflow can't use a sandbox, which is why there is no issue with loading header files.

image

@Kawanaao
Copy link
Contributor Author

Ah, I deleted the Yarn configuration where the nodedir was set and didn't replace it with npm. Now it makes sense.

@Kawanaao
Copy link
Contributor Author

I'm not quite sure, reh requires the vscode-ripgrep binary release, but should it be replaced with the system one?

@Kawanaao
Copy link
Contributor Author

It seems that reh cannot be installed offline for now.

@PF4Public
Copy link
Owner

It seems that the workflow can't use a sandbox, which is why there is no issue with loading header files.

Sandbox there is a little bit crippled, yes.

I'm not quite sure, reh requires the vscode-ripgrep binary release, but should it be replaced with the system one?

Would it work with the system one?

It seems that reh cannot be installed offline for now.

What makes you think that way? Maybe we miss some additional dependencies?

@Kawanaao
Copy link
Contributor Author

Kawanaao commented Oct 20, 2024

Would it work with the system one?

System ripgrep works great, use the new version of VSCode myself.

What makes you think that way? Maybe we miss some additional dependencies?

reh requires loading node-20. As I saw in the ebuild, there was a TODO to implement support for the system node, but for now, it still requires loading node.

https://github.com/microsoft/vscode/blob/main/build/gulpfile.reh.js

BUILD_TARGETS.forEach(({ platform, arch }) => {
	gulp.task(task.define(`node-${platform}-${arch}`, () => {
		const nodePath = path.join('.build', 'node', `v${nodeVersion}`, `${platform}-${arch}`);

		if (!fs.existsSync(nodePath)) {
			util.rimraf(nodePath);

			return nodejs(platform, arch)
				.pipe(vfs.dest(nodePath));
		}

		return Promise.resolve(null);
	}));
});
...
['reh', 'reh-web'].forEach(type => {
	const bundleTask = task.define(`bundle-vscode-${type}`, task.series(
		util.rimraf(`out-vscode-${type}`),
		optimize.bundleTask(
			{
				out: `out-vscode-${type}`,
				esm: {
					src: 'out-build',
					entryPoints: [
						...(type === 'reh' ? serverEntryPoints : serverWithWebEntryPoints),
						...bootstrapEntryPoints
					],
					resources: type === 'reh' ? serverResources : serverWithWebResources,
					fileContentMapper: createVSCodeWebFileContentMapper('.build/extensions', type === 'reh-web' ? tweakProductForServerWeb(product) : product)
				}
			}
		)
	));

	const minifyTask = task.define(`minify-vscode-${type}`, task.series(
		bundleTask,
		util.rimraf(`out-vscode-${type}-min`),
		optimize.minifyTask(`out-vscode-${type}`, `https://main.vscode-cdn.net/sourcemaps/${commit}/core`)
	));
	gulp.task(minifyTask);

	BUILD_TARGETS.forEach(buildTarget => {
		const dashed = (str) => (str ? `-${str}` : ``);
		const platform = buildTarget.platform;
		const arch = buildTarget.arch;

		['', 'min'].forEach(minified => {
			const sourceFolderName = `out-vscode-${type}${dashed(minified)}`;
			const destinationFolderName = `vscode-${type}${dashed(platform)}${dashed(arch)}`;

			const serverTaskCI = task.define(`vscode-${type}${dashed(platform)}${dashed(arch)}${dashed(minified)}-ci`, task.series(
---->			gulp.task(`node-${platform}-${arch}`),
				util.rimraf(path.join(BUILD_ROOT, destinationFolderName)),
				packageTask(type, platform, arch, sourceFolderName, destinationFolderName)
			));
			gulp.task(serverTaskCI);

			const serverTask = task.define(`vscode-${type}${dashed(platform)}${dashed(arch)}${dashed(minified)}`, task.series(
				compileBuildTask,
				compileExtensionsBuildTask,
				compileExtensionMediaBuildTask,
				minified ? minifyTask : bundleTask,
				serverTaskCI
			));
			gulp.task(serverTask);
		});
	});
});

@Kawanaao
Copy link
Contributor Author

Kawanaao commented Nov 1, 2024

Well... I think that using npm cache add is a mistake, this mechanism is too closed off, npm is very focused on cache proper functioning. Any modifications lead to issues with incorrect resolution of dependent packages, even when installing tgz files. I think it is worth returning to a local mirror

@Kawanaao
Copy link
Contributor Author

Kawanaao commented Nov 1, 2024

Project Shrinkpack looks quite interesting, it replacing online dependencies with their offline tar versions, which should be helpful. I think it would be enough to make a few adjustments to support tar.gz in a specific folder

@Kawanaao
Copy link
Contributor Author

Kawanaao commented Nov 2, 2024

No thanks, it's not tasty at all

TAR_BAD_ARCHIVE: Truncated input
npm warn tar zlib: Array buffer allocation failed
npm silly tar ZlibError: zlib: Array buffer allocation failed
npm silly tar     at Unzip.write (/usr/lib64/node_modules/npm/node_modules/minizlib/index.js:154:22)
npm silly tar     at Unpack.write (/usr/lib64/node_modules/npm/node_modules/tar/lib/parse.js:417:57)
npm silly tar     at [emitData] (/usr/lib64/node_modules/npm/node_modules/minipass/dist/commonjs/index.js:799:24)
npm silly tar     at Minipass.emit (/usr/lib64/node_modules/npm/node_modules/minipass/dist/commonjs/index.js:759:37)
npm silly tar     at Minipass.write (/usr/lib64/node_modules/npm/node_modules/minipass/dist/commonjs/index.js:376:18)
npm silly tar     at [emitData] (/usr/lib64/node_modules/npm/node_modules/minipass/dist/commonjs/index.js:799:24)
npm silly tar     at ReadStream.emit (/usr/lib64/node_modules/npm/node_modules/minipass/dist/commonjs/index.js:759:37)
npm silly tar     at ReadStream.emit (/usr/lib64/node_modules/npm/node_modules/fs-minipass/lib/index.js:173:22)
npm silly tar     at ReadStream.write (/usr/lib64/node_modules/npm/node_modules/minipass/dist/commonjs/index.js:376:18)
npm silly tar     at [_handleChunk] (/usr/lib64/node_modules/npm/node_modules/fs-minipass/lib/index.js:141:19)

Npm it just doesn't want to digest tgz, it's simply not capable of decompressing them in such quantities, but in order to build a dependency tree, you need to load all tgz, although the memory usage is low, but for some reason it completely ignores the memory and some errors constantly occur inside node like

' at Proxy.<anonymous> (/usr/lib64/node_modules/npm/bin/npm-cli.js:12:28)\n' +
' at /usr/lib64/node_modules/npm/ lib/cli/exit-handler.js:169:21\n' +
' at afterWrite (node:internal/streams/writable:708:5)\n' +
' at afterWriteTick (node:internal/streams/writable:694 :10)\n' +
' at process.processTicksAndRejections (node:internal/process/task_queues:81:21)',

and it seems like this signals correct reading.. but.. tar is damaged, tgz may be too , But and it doesn't want to use memory either.. and when reading correctly node returns 1 which terminates npm without any error message

@Kawanaao
Copy link
Contributor Author

Kawanaao commented Nov 7, 2024

It seems that, moreover, VSCode requires the presence of a .git folder. While using SSH-remote, I noticed that validation fails because, instead of the commit hash, VSCode uses the version from package.json as a fallback

@PF4Public
Copy link
Owner

VSCode requires the presence of a .git folder. […] validation fails because, instead of the commit hash, VSCode uses the version from package.json as a fallback

I think I've solved that via

einfo "Editing build/lib/getVersion.js"
sed -i '/.*\!version.*/{s++if \(false\)\{+;h};${x;/./{x;q0};x;q1}' \
build/lib/getVersion.js || die
and
export BUILD_SOURCEVERSION="${COMMIT_ID}"

@PF4Public PF4Public linked an issue Nov 10, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

app-editors/vscode-9999: fails to build; yarn not supported
2 participants