Load Pyodide runtime through HTTP

[garrettgu10]

This loads the Pyodide bundle through HTTP. It does this in a synchronous way, by creating a new kj::Thread and immediately disowning it, causing the current thread to be blocked on the completion of the new thread.

When a disk cache directory is provided at the command line using pyodide-bundle-disk-cache, the directory is used to cache pyodide bundles.

Existing test cases have been edited to use the pyodide.capnp.bin file generated through bazel, so changes in src/pyodide will be reflected instantly if tested through bazel test .... Otherwise, the correct version according to compat date will be fetched from the internet.

Here's a plan for how to test the disk cache writing (reading is covered by bazel tests) functionality:

mkdir /tmp/pyodide
./bazel-bin/src/workerd/server/workerd serve samples/pyodide/config.capnp --experimental --pyodide-bundle-disk-cache-dir=/tmp/pyodide --verbose
# should print loading pyodide package from internet
./bazel-bin/src/workerd/server/workerd serve samples/pyodide/config.capnp --experimental --pyodide-bundle-disk-cache-dir=/tmp/pyodide --verbose
# should no longer print loading pyodide package from internet & just work

[garrettgu10]

One point of jank is that the tip-of-tree pyodide bundle is still named pyodide-2.capnp.bin since the version number is still hardcoded in workerd. Ideally we want this to be named pyodide-latest.capnp.bin or something similar but this would require us to finish the story on compat-date -> pyodide version mapping.

[garrettgu10]

Another thing we need to figure out before merging -- how can we determine if a worker is a Python worker or not? We need to know this so that we don't fetch the Pyodide bundle when the worker doesn't need it.

[hoodmane]

Since it's blocking, why not call fetchPyodideBundle in workerd-api just before it's needed?

[dom96]

finish the story on compat-date -> pyodide version mapping.

This story is complete with #2464. You should be able to use it to determine the bundle filename to download.

how can we determine if a worker is a Python worker or not?

By checking the WorkerBundle. We've got the hasPythonModule for this in EW which you should be able to recreate here.

[garrettgu10]

@dom96 ah, makes sense. We should probably copy pyodide-2.capnp.bin to pyodide-0.26.0a2.capnp.bin on our bucket so we can use that. I'll do that

[dom96]

LGTM but I think you should integrate PythonSnapshotRelease into here before merging. You probably want to move getPythonSnapshotRelease out of EW and into workerd, then use it here to get the right Python version to load.

[hoodmane]

Could you split "refactor bazel test setup" out into a separate PR?

[hoodmane]

We'll want to select between "dev" and the appropriate version by setting an appropriate compat flag so that tests can be a bit more declarative. I think it would be confusing if you try to test a specific version but the test uses latest because it exists. But we can deal with that in a followup.

[hoodmane]

I think it would be tidier to have fetchPyodideBundle return a kj::Maybe<> directly, but we can tidy up that sort of thing in followup PRs.

[hoodmane]

Thanks @garrettgu10! Let's merge this.