Add Simple Nix Flake

[JosephGoulden]

This is new PR to add a simple flake.nix to this project. It will allow downstream projects to easily import CLN into their system. They will be able to get new releases as soon as they are available instead of having to wait for nixpkgs to be updated. And they will be able to more easily test any changes being made in CLN.

There is an existing PR #6650 but progress has stalled with getting the Python dependencies working. I think it is best to leave that for a later date and focus on a minimal working and useful flake which I have created in this PR.

The default package is called cln, it has been copied from the one currently in nixpkgs. The derivation calls the Makefile and builds the binaries (lightningd, lightning-cli, lightning-hsmtool, reckless).

There is also a package called rust which builds the Cargo workspace at the root of the project. Binaries are then copied from here into the build directory of the cln package.

The use of git submodules makes the UX a little awkward but still works as expected with the right commands. e.g;

• nix develop will create the default shell env with the build and runtime dependencies of the cln package. Then you can call make manually and the project will compile successfully.

• nix develop .#rust will create a shell env for developing rust.

• nix build .?submodules=1 will build the default package (cln).

• nix flake check .?submodules=1 will build the cln and rust packages. Rust tests are run during the build. There are also checks to run cargo audit and nixfmt.

• If you have nix installed you can run use nix run "git+https://github.com/hashrelay/lightning?ref=flake&submodules=1#lightningd" to run lightningd without having to manually clone the repo. This make use of the flake output apps.

I've added a new .version file to source control. Having the version is source control is required in order to make a pure nix flake (fully deterministic, reproducible builds without dependencies on external environment). I understand this may not be ideal (from what I understand the version is usually derived from the git tags). But I can also see that the current version is already specified in multiple python files that must be updated each release. So the new .version file is one more place to update (perhaps it can become the only place to update). The version could also be put straight into default.nix if .version is not wanted.

I've added a new target to the CI workflow that checks the nix package builds successfully.

[chrisguida]

Hmm, for some reason it's not working for me on Ubuntu

$ nix develop
error:
       … while updating the lock file of flake 'git+file:///home/cguida/work/lightning/elementsproject?ref=refs/heads/pr7656&rev=0fc6bdf6d11a8a2a33567e480f0f7207a0172edc'

       error: unsupported tarball input attribute 'lastModified'
(cln-meta-project-py3.12) 2024-09-16 12:45:22 cguida@cg-acer:~/work/lightning/elementsproject$ nix --version
nix (Nix) 2.17.0

got any ideas?

[chrisguida]

ok i ran sudo -i nix upgrade-nix which upgraded my nix

$ sudo -i nix upgrade-nix
[sudo] password for cguida: 
replacing old 'nix-2.17.0'
installing 'nix-2.18.5'
building '/nix/store/yrgxb0lilsz84d1m8sllqq7pf6mxn5vr-user-environment.drv'...

Now it works!!

New problem though, the Rust build is failing with a giant pile of errors:

Any ideas on that?

[JosephGoulden]

All those errors stem from the "corrupt metadata" of lazy_static. I've not seen that before, can you try deleting your .cargo/registry, or at least that crate. And run cargo clean as well.

[chrisguida]

I ran rm -rf ~/.cargo/registry and now it builds!

$ lightningd/lightningd --version
v24.08-11-g0fc6bdf-modded

Seems like maybe the dev shell should handle this though, don't you think?

[chrisguida]

nix build .?submodules=1 works too!! not sure if this would have worked if I hadn't deleted the cargo registry though.

[vincenzopalazzo]

I ran rm -rf ~/.cargo/registry and now it builds!

This does not sounds right? the all point of nix is to have a sandbox, so ~/ should not be your home

[JosephGoulden]

I ran rm -rf ~/.cargo/registry and now it builds!

This does not sounds right? the all point of nix is to have a sandbox, so ~/ should not be your home

This is because I have not written derivations specifically for the cargo builds yet. I can do that later using crane if desired. At the moment the cargo builds are done by the Makefile with the cargo nativeBuildInput. It's still an improvement on the current situation as everyone using the flake will have the same version of cargo.

[JosephGoulden]

Hi @ShahanaFarooqui I'm not sure how to progress this PR. I can see you have done the past few releases. Would you be able to comment please? In particular about the .version file I proposed.

[ShahanaFarooqui]

@JosephGoulden Sorry, I am not familiar with Nix currently, so I might not be helpful here.

But as for this PR, with @chrisguida and @vincenzopalazzo reviewing, it is already in good hands.

Regarding the .version file, my suggestion might not apply in this situation, but just wanted to propose using the existing ./version_gen.h instead of adding another file for the same purpose (if that makes sense).

[chrisguida]

@JosephGoulden I think you should see if you can incorporate crane or some other framework to make sure the cargo deps are reproducible.

Lmk if you want to pair on it, I've never built anything on crane but I have some examples that other people have written that could maybe help.

Another option would be to simply disable rust since it's only needed for grpc atm, though there is also a new rust rest plugin that will hopefully be included as a built-in soon.

If you think it will take too long, we can still merge this and maybe we can get more eyes on it :)

[chrisguida]

For instance this project uses naersk and it works really well

https://github.com/chrisguida/smaug/blob/update-upstream/flake.nix

[chrisguida]

Here's another example from @RCasatta in the sling plugin daywalker90/sling@11abe1b

[JosephGoulden]

@chrisguida I added a crane derivation to build the cargo workspace. The cln-grpc binary gets copied into the plugins directory by the postInstall phase of the default package. It will be easy to add the new rest plugin when that's ready.

As you know there is lots more we could do with nix but I think this is a good start.

[JosephGoulden]

@JosephGoulden Sorry, I am not familiar with Nix currently, so I might not be helpful here.

But as for this PR, with @chrisguida and @vincenzopalazzo reviewing, it is already in good hands.

Regarding the .version file, my suggestion might not apply in this situation, but just wanted to propose using the existing ./version_gen.h instead of adding another file for the same purpose (if that makes sense).

Thanks @ShahanaFarooqui. I did consider trying to reuse version_gen.h but really I think its better to have a .version file with only the version in it. Then other tools can use that to generate their required format.

[chrisguida]

Hmm build still failing. Maybe i need to update the submodules?

[chrisguida]

Ok that seems to have worked...

Now i get

It actually does work, just prints out a weird warning about clnrest.

Anyway cool, I'm good with this as long as there's documentation somewhere to tell the user to manually update the submodules by running git submodule update (maybe --init --recursive too?)

[chrisguida]

The Rust build does seem to take very long and makes my computer really hot! But at least it works 🚀