Proposing another implementation of lisp-modules #155851
Comments
|
Copying manually the generated definitions for anything that depends on them sounds pretty bad (in the context of Nyxt being packaged like a non-Quicklisp package that depends on large amounts of whatever)… Is compilation supposed to happen for all the stuff in withPackages at once? |
|
yeah it's all pre-compiled at once when it comes to compilatioin, there was another problems that I had:
|
|
I haven't had problems with copying generated definitions. They are almost the same as the manually specified ones, with the exception that when building a system like Now that I think about it it would have to be the same for the manual package, but I haven't had such a package yet. So in that case the ql imported things is just a way to save a lot of typing. |
|
I have nothing against some special-case workarounds; I somewhat like the idea of binary cache compatibility, though. About copying: it's trivial, but it needs some integration with mass updates for manual packages with tons of dependencies. |
|
Hmmm I understand, that is problematic interaction between the imported and manual packages is kind of tricky Maybe I could load the quicklisp packages without rewriting their dependency trees at first, then load the manual packages and provide quicklisp imports as additional libraries. Just to break the current inter-dependency. Then go through the resulting thing and do whatever such as prefer manual libs to ql. Then build that derivation |
|
Maybe a generator could do all the copying before use, though. |
|
Are you thinking about a separate script that updates the file containing the manual packages? That could work . Or even a separate file, "ql-deps.nix" or something to keep the manual file clean |
|
Basically how the current Lisp infrastructure just generates the Quicklisp data once as a nix file, yes. |
Added that in Uthar/nix-cl@a0eb90a It was pretty productive because I could remove most of my package code and just reuse the QL generated things. So thanks a lot for this suggestion @7c6f434c! For example, to pin the version of this code in Later on, it will simply pick up updates to its dependencies from the quicklisp generated file. At the same time, anything that depends on Though now it's redundant with the |
|
Though now it's redundant with the `extras` mechanism in `ql.nix`. Meaning *some* customizations can be done in either `ql.nix` or `packages.nix`. Hmmm not sure what to do about that. Myself I would keep non-QL packages in `packages.nix`, and trivial fixes to QL packages (addded native library etc.) in `ql.nix`.
Which is also more or less what is done now (which is surely a good thing)
Do I understand correctly that you trust Quicklisp dependency data?
(Hmm, now if only I understood how to combine your code with binary-cache-friendlyness/precompilation, it could indeed be attractive to just replace lispPackages with that)
|
|
I do directly read the quicklisp dependency data, then do some merging on it in the importer to capture dependencies of dependencies etc. Most stuff that I tried building this way just worked - in case of problems a common one was a dependency on a "slashy" system. I ignore those in the importer because to it, a unit of granularity is the system, not the One of the cool things that I'm thinking about would be the ability to run Nix builds of the entire Quicklisp on Nix orgs' Hydra servers - probably a gazillion bugs would be uncovered. At least for sbcl/ccl where they have pretty fast compilers (?) Not really sure how that would look from resoures/political side. Could you expand some more about the precompilation concept? If the concern is recompiling all libs for each wrapper, then that's not the case - the lisp packages themselves get compiled once ( |
|
My current workaround for slashy systems is to add them to the |
|
I do directly read the quicklisp dependency data, then do some merging on it in the importer to capture dependencies of dependencies etc.
Most stuff that I tried building this way just worked - in case of problems a common one was a dependency on a "slashy" system. I ignore those in the importer because to it, a unit of granularity is the system, not the `.asd` file (that's why these files are created as necessary) - but "slashy" systems belong to the "root-name-part.asd" file. At the time I didn't know how to handle that, but maybe there is a way.
There have been Quicklisp releases in the past with some more interesting problems in dependency data (which is where quicklisp-to-nix takes its motivation).
One of the cool things that I'm thinking about would be the ability to run Nix builds of the entire Quicklisp on Nix orgs' Hydra servers - probably a gazillion bugs would be uncovered. At least for sbcl/ccl where they have pretty fast compilers (?) Not really sure how that would look from resoures/political side.
Even now CCL doesn't build 100% of lispPackages. SBCL might be doable. Quicklisp is actually somewhat small, so running everything on SBCL will probably end up doable even locally. And as we monotonically grow the set of packages we grab, just maxing it would not be that much of a surprise…
I don't think we'd uncover any meaningful bugs, because entire Quicklisp is built before release and a package needs to build at least somewhere to not be removed. We would get a somewhat better tool for impact measurement for SBCL upgrades, though.
Could you expand some more about the precompilation concept? If the concern is recompiling all libs for each wrapper, then that's not the case - the lisp packages themselves get compiled once (`sbclPackages`), then any lisp wrappers (`sbclWithPackages`) reuse that without recompiling anything (they just put together a wrapper shell script). Sorry, maybe I didn't understand your previous question correctly so the answer could be misleading.
Ooh, that's better, I misunderstood you, sorry. I interpreted some of the «all together» as compilation of the packages only once they are all gathered together into an environment.
|
|
My current workaround for slashy systems is to add them to the `systems` list of the thing that is being depended on. So, `nyxt` depends on `iolib/pathname` then I set `iolibs.systems` to ` [ "iolib" "iolib/pathname" ]`
Presumably, if you already grab the tarballs in the generator, you can as well unpack them and obtain the list of such systems, right?
|
That's interesting. I only imported the QL releases 2021-12-30 and 2021-08-07 (that one had a bug with a missing dependency of
Hmm you're right. I could actually try building the whole of QL locally using SBCL (Currently that's 4310 packages). Or maybe start with the subset that's in
That could be an option, but Quicklisp's systems.txt already contains those systems in the third column. I just skipped them for now for lack of a better idea. On the other hand, with 2021-12-30, The problem for me was to somehow make slashy systems work with
So in effect it "packs" a bunch of systems into a single There is the possiblity to simply always build every slashy subsystem of a system. But that could possibly add unnecessary dependencies in cases the slashy systems won't be used. Because real libraries use the |
|
OK, I'll build the systems in |
|
I think the massive numbers of missed dependencies were quite long ago now. slashy systems have yet another annoying habit of creating circular dependencies between .asd files, yeah. |
|
Results for Gonna do the same for things in |
|
From a glance, looks about right. |
|
Status of Things that don't build are the slashy systems: And, additionally for CCL, three packages that are explicitly SBCL-only Plus I would propose another solution to the slashy systems problem. On branch Here's a This could even be used when a certain package depends on some slashy systems, to provide the dependency in an "anonymous" way - without cluttering the original Given their "optionality" nature this seems pretty reasonable to me - what do you think? |
|
Got a question about this line |
|
This could even be used when a certain package depends on some slashy systems, to provide the dependency in an "anonymous" way - without cluttering the original `sbclPackages.cl-postgres`.
Given their "optionality" nature this seems pretty reasonable to me - what do you think?
I think there are completely optional (e.g. tests) things like that that seem indeed just skippable, a few places where there is no circular dependencies involved, and I think one or two loops that I somewhat hate where downstream-useful libraries are locked in a circular between-asd dependency. But we don't handle that currently anyway.
|
|
Hm, we do not seem to need this now. Does building an older version still need it in your setup? |
|
I get the error with |
|
Hm, probably not DE-specific (not sure what their sandbox wants though) |
|
To be precise, this is what happens:
|
|
Ah, that might be dependent on having something installed in the profile, possibly via DE in system path… Re: slashy systems — I think current generation logic defaults to including these systems and blocking the problematic ones. Maybe it's better to keep that approach? |
|
Posting rest of CL implementations for completeness ( |
Tested it a little bit, here are some results. I used: lispPackages_new.sbclWithPackages (p:
[ p.cl-async-base
p.cl-async-util
p.cl-async
])as an example, because all three contain the It breaks though when trying to load
|
|
It breaks though when trying to load `cl-async` two times in a row, which is... weird:
```
* (require 'asdf)
("ASDF" "asdf" "UIOP" "uiop")
* (asdf:load-system 'cl-async)
T
* (asdf:load-system 'cl-async)
debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {1004B88463}>:
Error opening #P"/nix/store/x541kcj7sgv46vj5vr7832i76f3jz7km-cl-async-base-20211020-git/src/util/package-tmpGHU3ALSV.fasl":
Read-only file system
0]
```
ASDF plan timestamps are slightly magical, and maybe cl-async does something during the load that ends up throwing off ASDF to decided the build plan is fresher than the output (even if _source files_ are older)
|
|
Just tried this in 22.05. Besides the fact, that cl-async won't load the second time, it also breaks the same way when I |
|
I think require passes through to ASDF so it is not surprising the results are the same… |
|
It does, but it won't load cl-async even once unlike load-system |
|
It does, but it won't load cl-async even once unlike load-system
Hm, strange indeed…
|
|
I'm having problems with Demo of failing build: https://github.com/hraban/nix-lisppackages-new-demo Is this how you're supposed to use it? Specifically it's package Full build error: The problem seems to be that cl-async.asd defines cl-async-base and cl-async-util, the latter including the former, but somehow when something (?) in the final build step tries to load cl-async-util, it tries to load it from cl-async-base's dir, where cl-async-util hasn't been compiled, which causes it to try and create the .fasl there, which fails. I don't really know where to go from here, unfortunately. My system: In my local nixpkgs-master checkout, I have cherry-picked two extra commits to make sbcl work: Thought I'd share this here as this case study might be of interest in the larger scheme of things. Happy to move it to a separate issue. |
|
Update: I think I fixed it. Could anyone interested, please have a look and tell me if this is sane? #196818 |
|
@Uthar looking at imported.nix, it seems individual packages are split up into their constituent systems, each available as top-level packages in nix. E.g. It seems to be these packages in particular that cause grief, so I was wondering what the rationale was for making them all top-level, nix managed dependencies. |
|
Hi @hraban |
|
Try like this --- a/default.nix
+++ b/default.nix
@@ -13,9 +13,4 @@ let
lisp = "${pkgs.sbcl}/bin/sbcl --script";
dontStrip = true;
};
-in
-asdf-only.overrideAttrs (final: prev: {
- buildPhase = prev.buildPhase + ''
- ${prev.lisp} build.lisp
- '';
-})
+in asdf-only
|
I made systems exist in their own I think that asdf started warning about this some time ago, which caused a huge drama |
|
in particular to work around this problem:
|
|
cl-async-base and cl-async-util are both defined in cl-async.asd, not in their own .asd files. They're "private" packages, internal to cl-async, and shouldn't really be exposed as top-level. See the discussion in #196818 . |
|
yeah that could be better because of less source duplication, but it was easier to write quicklisp importing for me like this |
|
because of less special cases |
|
What do you think if it was possible to use Lisp packages like this?
|
|
I'm also thinking about something like this in let me know of any thoughts |
|
Now that I think about the slashy systems problem, if there is a way to To me it seems that asdf is simply hard-coded to compile every |
|
What |
|
It could just load the files, there could be |
|
I wonder if the most proper solution available would be just to provide ASDF a function for compiled output location that tries next to the source file, in the current build output, and in $HOME-stored cache in that order… |
|
Recently I read the python infrastructure in nixpkgs. Now what I'm thinking to do is to make secondary systems optional via overlays. So that we can still keep the fasls in /nix/store. {
# Instead of:
sbcl-with-foo-and-bar = sbcl.withPackages (ps: [ ps.foo ps.bar_slash_baz ]);
# Use overlays:
sbcl' = sbcl.override {
packageOverlays = self: super: {
bar = super.bar.overrideLispAttrs (old: {
systems = old.systems ++ [ "bar/baz" ];
});
};
};
sbcl-with-foo-and-bar = sbcl'.withPackages (ps: [ ps.foo ps.bar ]);
} |
|
see #218870 |
Hi!
I made a small nix library (
nix-cl), for working with ASDF and Quicklisp, which has been working great for me recently. The code is still a mess but I want to show it to you and see if it sparks interest.Basically I think it could replace the current
lisp-modulesand provide a number of benefits.If you'd like to test it out you should use
direnvor source.envrcto get the examples working. (Have to set NIX_PATH).(Also worth noting, I think that
nixpkgsdoesn't provide Clasp but it worked with my own version.)I had these problems with
lisp-modules- please correct me if I'm wrong here:common-lisp.sh) - no documentation$HOMECommon Lisp compilation files (fasl) are not being used #142604.jarin theCLASSPATH)nix-shellbecause a lisp-with-packages is achieved by putting the libraries themselves inbuildInputsand running the genericcommon-lisp.sh.asdis not in the root directory (e.g. jzon)Which are improved in
nix-cl:${lisp}WithPackagesCL_SOURCE_REGISTRY,ASDF_OUTPUT_TRANSLATIONSare set in a granular way, per packageabclWithPackages/cclWithPackages/eclWithPackages/sbclWithPackagesetc. makes it possible to have multiple lisps in onenix-shellasd's not in root by using "//" suffix to CL_SOURCE_REGISTRY (cool feature of ASDF - manual)pname,version,src.There are issues with it still where I haven't had the need to fix them:
/share/java/*.jar(fromfetchMavenArtifact)_3d-vectors)@7c6f434c
The text was updated successfully, but these errors were encountered: