Towards tweaselORG/meta#36: Add auto-archiving script #54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
baltpeter
requested changes
Sep 25, 2023
zner0L
force-pushed
the
z_archive-script
branch
from
be8cee3 to
1fff917
Compare
baltpeter
requested changes
Sep 29, 2023
zner0L
force-pushed
the
z_archive-script
branch
from
85e22bf to
556de9d
Compare
baltpeter
requested changes
Oct 9, 2023
zner0L
force-pushed
the
z_archive-script
branch
from
556de9d to
a17891b
Compare
zner0L
force-pushed
the
z_archive-script
branch
from
ab78635 to
0ba388d
Compare
baltpeter
previously approved these changes
Oct 26, 2023
zner0L
force-pushed
the
z_archive-script
branch
from
0ba388d to
4f07270
Compare
This was surprisingly hard to do. Because nodejs caches imports, we need to somehow clear the import cache. This can be done naively (https://ar.al/2021/02/22/cache-busting-in-node.js-dynamic-esm-imports/#cache-invalidation-in-esm-with-dynamic-imports) for plain js files, but it leaks memory and fails for typescript files, since the lodaer doesn’t support it. So, we run each archiving task in a new worker thread that has its own import context. But, since the adapters we are importing are written in typescript, we need to register a loader in the worker context, because it doesn’t have it loaded by default (like the main thread does via `tsx`). Luckly, this feature has just been added to node, but we need to bump our node version to use it. Now we can dynamically import the all chnages to modules.
zner0L
force-pushed
the
z_archive-script
branch
from
4f07270 to
05c23e0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is based on #32, which needs to be merged first.