Containerize dependencies #92

har7an · 2022-02-24T08:08:35Z

This MR introduces a Containerfile that builds a container with all the doc dependencies (mdbook and hugo), and expands the watch-serve.sh to make use of this container if the deps aren't available locally.

It should work with both podman and docker as container runtimes, although I only have podman so I couldn't yet verify whether it really works with docker. If someone reading this has docker, I'd be thankful for feedback on that.

If neither the deps nor a container runtime can be found, a message is now printed to stdout to inform the user what's missing.

a-kenji · 2022-02-24T09:32:07Z

Thank you!
This is great work already, sadly this locks me out from how I use the script.

har7an · 2022-02-24T09:42:48Z

Phew, I must confess I never worked with nix before... Is there a way for me to tell if you're calling the script with nix? Does it set some env var? Or is this:

nix develop -v ...

A general nix command similar to how containers work? Then I could also have it check for the existence of nix and run this by default. Or does it come with side-effects?

a-kenji · 2022-02-24T09:47:56Z

Afaik when calling commands like that there is no way to tell that this script is called from nix.
I think checking for the existence of nix is enough. I am using flakes right now and the dependencies are specified here:
https://github.com/zellij-org/zellij-org.github.io/blob/main/flake.nix
but I can also add a compatibility layer in a bit that would make it also work without flakes.

har7an · 2022-02-24T09:50:22Z

Well I don't know about flakes but if you say that the existence of flake.nix here is enough to make it work with nix then I'll believe you and adapt the script in a bit. :)

har7an · 2022-02-24T10:39:40Z

Just to make sure I understand how nix works: Does nix develop -c also accept the command to run verbatim? Like so:

nix develop -c "{ mdbook watch docs/ -d ../static/documentation & hugo server; }"

Or isn't it meant to work that way?

a-kenji · 2022-02-24T11:32:55Z

Just to make sure I understand how nix works: Does nix develop -c also accept the command to run verbatim? Like so:
nix develop -c "{ mdbook watch docs/ -d ../static/documentation & hugo server; }"
Or isn't it meant to work that way?

It isn't meant to work that way.

a-kenji · 2022-02-24T11:38:09Z

I added the compatibilty, now anyone with nix installed can run the shell,
regardless if they have flakes installed.

har7an · 2022-02-24T12:26:19Z

I added the compatibilty, now anyone with nix installed can run the shell,
regardless if they have flakes installed.

Where exactly is that? I don't see it here. I'd really like to know how it works for future commitments, since nix seems to become increasingly popular.

har7an · 2022-02-24T12:33:13Z

Ah I see, you did it in a separate MR. Given that you use nix I assume you don't have docker, is that correct? I'll try to find someone with docker (Or find a way to run docker from podman) to test it there.

a-kenji · 2022-02-24T15:02:39Z

I added the compatibilty, now anyone with nix installed can run the shell,
regardless if they have flakes installed.

Where exactly is that? I don't see it here. I'd really like to know how it works for future commitments, since nix seems to become increasingly popular.

The compatibility is here:
https://github.com/zellij-org/zellij-org.github.io/blob/main/shell.nix

It basically just uses the flake.lock from the flake.nix to understand the pinned channels, kind of like nur would do it.

So instead of:

nix develop -c ./watch-serve.sh

one would use it like:

nix-shell --command ./watch-serve.sh

And both derivations would provide the same output.

Yes, I very seldom use docker and if, then I also use podman.

har7an · 2022-02-25T08:14:09Z

Okay so I setup a NixOS VM because I wanted to see first-hand how this works and think about a way to handle this proper, so your workflow hopefully doesn't break. Unfortunately just entering your command from above:

nix-shell --command ./watch-serve.sh

gives me an error about reaching the end of a string and not being able to find a file in the nix store (on the main branch, that is). Any ideas what's causing this?

a-kenji · 2022-02-25T10:45:57Z

What version is the nixvm? I don't really know what could cause it.

Do you mind trying the first command?

nix develop -c ./watch-serve.sh

Maybe your vm comes already with a version of nix that has stabilized flakes?

It could be that you would need to set up channels first for the vm,
if using nix-shell. But I would need to look into that.

Edit:

Does that mean that just running nix-shell does also raise this error?

har7an · 2022-02-25T10:51:44Z

The VM is version 22.05 (unstable). I did try the first command, and that didn't work without enabling some of the experimental features, flakes was one of them.

Oh well, I'll go on to testing it with docker then. I can confirm that the podman version works great, I wrote #95 with it.

a-kenji · 2022-02-25T10:53:22Z

What happens if you type:

 nix-channel --list && sudo nix-channel --list

har7an · 2022-02-25T11:01:16Z

Huh. The first output is empty, the second shows the nixos channel... But running as root doesn't work either.

a-kenji · 2022-02-25T11:10:53Z

Thanks for all the help so far already!

Do you mind trying out:

nix-channel --update && sudo nix-channel --update

And trying it again?

I have also added a new file, that shouldn't really make an impact on the shell
as far as I understand it, but it's worth a shot and could still help people building the shell manually.

a-kenji · 2022-02-25T11:11:54Z

Huh. The first output is empty, the second shows the nixos channel... But running as root doesn't work either.

Oh yeah, that could make sense, since it's a vm it proably doesn't need root.

har7an · 2022-02-25T13:18:44Z

Do you mind trying out:

I did. I now switched to running nix from podman because its less heavy (obviously), but the result is still the same. Here's the output with trace, if you want to have a look:

Output of "nix-shell --command ./watch-serve.sh"

# nix-shell --show-trace --command ./watch-serve.sh
error: renaming '/nix/store/add-241-2/x' to '/nix/store/1jw9glil35sdhsxwpw754rxc9b0s490w-source'

       … while fetching the input 'git+file:///root/zellij-org.github.io'

       … while evaluating 'tryFetchGit'

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:92:19:

           91|     # tree is a valid git repository.
           92|     tryFetchGit = src:
             |                   ^
           93|       if isGit && !isShallow

       … from call site

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:104:42:

          103|     { lastModified = 0; lastModifiedDate = formatSecondsSinceEpoch 0; }
          104|       // (if src ? outPath then src else tryFetchGit src);
             |                                          ^
          105|

       … while realising the context of a path

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:142:19:

          141|
          142|           flake = import (sourceInfo + (if subdir != "" then "/" else "") + subdir + "/flake.nix");
             |                   ^
          143|

       … while evaluating anonymous lambda

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:133:13:

          132|     builtins.mapAttrs
          133|       (key: node:
             |             ^
          134|         let

       … from call site

       … while evaluating the attribute 'shellNix'

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:194:5:

          193|
          194|     shellNix =
             |     ^
          195|       defaultNix

Output of "nix develop -c ..."

# nix --extra-experimental-features nix-command --extra-experimental-features flakes --show-trace develop -c ./watch-serve.sh
error: renaming '/nix/store/add-261-1/x' to '/nix/store/1jw9glil35sdhsxwpw754rxc9b0s490w-source'

       … while fetching the input 'git+file:///root/zellij-org.github.io'

a-kenji · 2022-02-25T19:38:18Z

Do you mind trying out:

I did. I now switched to running nix from podman because its less heavy (obviously), but the result is still the same. Here's the output with trace, if you want to have a look:
Output of "nix-shell --command ./watch-serve.sh"
Output of "nix develop -c ..."

Thank you! That was very helpful. I updated the dependencies again, do you mind trying it with the latest main? I am quite confident that both commands should work for you now.

har7an · 2022-02-28T08:49:53Z

I am quite confident that both commands should work for you now.

Well I can report a partial success now:

output of "nix-shell --command ./watch-serve.sh" on fresh nix


# nix-shell --command ./watch-serve.sh 
error: getting status of '/nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source': No such file or directory

output of "nix develop -c ./watch-serve.sh" (Yaaay!)


# nix develop --extra-experimental-features nix-command --extra-experimental-features flakes -c ./watch-serve.sh 
2022-02-28 08:43:49 [INFO] (mdbook::cmd::watch): Listening for changes...
Start building sites … 
hugo v0.92.2+extended linux/amd64 BuildDate=unknown

                   | EN   
-------------------+------
  Pages            |  18  
  Paginator pages  |   0  
  Non-page files   |   0  
  Static files     | 119  
  Processed images |   0  
  Aliases          |   4  
  Sitemaps         |   1  
  Cleaned          |   0  

Built in 104 ms
Watching for changes in /zellij-org.github.io/{archetypes,content,static,themes}
Watching for config changes in /zellij-org.github.io/config.toml
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

output of "nix-shell --command ./watch-serve.sh" after "nix develop ..."


# nix-shell --show-trace --command ./watch-serve.sh 
error: renaming '/nix/store/add-355-1/x' to '/nix/store/d7x29d79gws2yw043hmyi082hvjzs67z-d9wpzgyjlpw1424kw2lyilah6690fnk3-source'

       … while adding path '/nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source'

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:46:19:

           45|     else if info.type == "path" then
           46|       { outPath = builtins.path { path = info.path; };
             |                   ^
           47|         narHash = info.narHash;

       … while realising the context of a path

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:142:19:

          141|
          142|           flake = import (sourceInfo + (if subdir != "" then "/" else "") + subdir + "/flake.nix");
             |                   ^
          143|

       … while evaluating anonymous lambda

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:133:13:

          132|     builtins.mapAttrs
          133|       (key: node:
             |             ^
          134|         let

       … from call site

       … while evaluating anonymous lambda

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:145:25:

          144|           inputs = builtins.mapAttrs
          145|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                         ^
          146|             (node.inputs or {});

       … from call site

       … while evaluating the attribute 'devShell'

       at /nix/store/s94crl5bwxazls6xr6pfa1qjgpjyi7v7-source/flake.nix:12:9:

           11|       rec {
           12|         devShell = pkgs.mkShell {
             |         ^
           13|         buildInputs = [

       … while evaluating the attribute 'devShell.x86_64-linux'

       at /nix/store/0xrzw1pz5rkq1qxdbszxl5j8xy4fdvha-source/default.nix:111:26:

          110|                   then (pushDownSystem system (attrs.hydraJobs or {}) ret.hydraJobs)
          111|                   else { ${system} = ret.${key}; };
             |                          ^
          112|             in attrs //

The nix develop one works now, but I have no clue what the other commands dislike...

har7an · 2022-02-28T08:52:02Z

Fun fact: It seems that the nixos/nix container has so many layers that sometimes podman struggles to determine the image size. Had to delete the container to restore podman to working order. ^^

a-kenji · 2022-02-28T08:56:38Z

@har7an,
thank you for checking again!
Did you run nix channel --update before running the nix-shell one?

Edit: flakes don't rely on channels the way the normal nix command does, that way they are much more robust.

har7an · 2022-02-28T09:01:59Z

Here's the exact order of commands I executed:

❯ podman run --rm -it docker.io/nixos/nix
bash-4.4# git clone https://github.com/zellij-org/zellij-org.github.io.git
Cloning into 'zellij-org.github.io'...
remote: Enumerating objects: 2003, done.
remote: Counting objects: 100% (66/66), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 2003 (delta 31), reused 38 (delta 20), pack-reused 1937
Receiving objects: 100% (2003/2003), 22.39 MiB | 9.86 MiB/s, done.
Resolving deltas: 100% (1323/1323), done.
bash-4.4# nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable
bash-4.4# nix-channel --update
unpacking channels...
bash-4.4# cd zellij-org.github.io/
bash-4.4# nix-shell --show-trace --command ./watch-serve.sh
error: getting status of '/nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source': No such file or directory

       … while adding path '/nix/store/d9wpzgyjlpw1424kw2lyilah6690fnk3-source'

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:46:19:

           45|     else if info.type == "path" then
           46|       { outPath = builtins.path { path = info.path; };
             |                   ^
           47|         narHash = info.narHash;

       … while realising the context of a path

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:142:19:

          141|
          142|           flake = import (sourceInfo + (if subdir != "" then "/" else "") + subdir + "/flake.nix");
             |                   ^
          143|

       … while evaluating anonymous lambda

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:133:13:

          132|     builtins.mapAttrs
          133|       (key: node:
             |             ^
          134|         let

       … from call site

       … while evaluating anonymous lambda

       at /nix/store/x8iqw6zwhvw2agp3f6zbpd0xij3249kq-source/default.nix:145:25:

          144|           inputs = builtins.mapAttrs
          145|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                         ^
          146|             (node.inputs or {});

       … from call site

       … while evaluating the attribute 'devShell'

       at /nix/store/s94crl5bwxazls6xr6pfa1qjgpjyi7v7-source/flake.nix:12:9:

           11|       rec {
           12|         devShell = pkgs.mkShell {
             |         ^
           13|         buildInputs = [

       … while evaluating the attribute 'devShell.x86_64-linux'

       at /nix/store/0xrzw1pz5rkq1qxdbszxl5j8xy4fdvha-source/default.nix:111:26:

          110|                   then (pushDownSystem system (attrs.hydraJobs or {}) ret.hydraJobs)
          111|                   else { ${system} = ret.${key}; };
             |                          ^
          112|             in attrs //

a-kenji · 2022-02-28T12:07:50Z

@har7an,
Thank you!
Do you mind testing it out again from main? I added another update.

har7an · 2022-02-28T18:40:10Z

@a-kenji
Sure thing, happy to help. :)
The message is different now, but still no success:

# nix-shell --show-trace --command ./watch-serve.sh
error: renaming '/nix/store/add-53-5/x' to '/nix/store/6lk1r8frx7rw1spfbfj2hi88fb853w17-source'

Entered the same commands as above.

a-kenji · 2022-02-28T19:18:59Z

Ok, I give up now with the compatibility.
There must be a specific issue between flake-compat, flakes and this docker container. Is it rebuilt frequently? I am usually fixing the issue and once you have this issue when I try it out I have exactly the same issue as you!

Now I added a shell.nix manually, which doesn't have it's dependencies pinned - so it should get them from the channel. If you would be so kind and check if the command also works on your side?

that includes the doc dependencies. Builds a container of about 145 MB that builds and serves the docs locally.

for people that have a container runtime available and prefer not to install the dependencies locally. If the dependencies are installed locally, it will call these directly instead of using the container. Otherwise it will check if podman is installed and use that, or docker otherwise, build the container from the `Containerfile` and run it with the project directory mounted. If neither the dependencies nor a suitable container runtime can be found, a messages is printed to stdout that tells the user what is needed to use the script.

Add the `--rm` flag to automatically delete the containers after they have been used. The script won't restart existing containers anyways, so there is no point in keeping them. It will only fill up the users disk.

har7an · 2022-03-01T07:19:00Z

Works like a charm. :)

But it looks to me as if the current version of hugo has some issues. Whenever I make it do something else than showing me its --help, it will segfault on some NULL references...

Calls the script with `nix-shell --command` if it detects that nix is present on the host.

har7an · 2022-03-01T07:21:51Z

@a-kenji If you want you can give the script itself another try now. If it detects the presence of nix-shell, it will now automatically reexecute the script with nix-shell --command instead, if hugo and mdbook weren't present.

a-kenji · 2022-03-02T07:49:09Z

@har7an,
yes it works for me, thank you!
Do you mind adding the first command too?

nix develop --command ./.sh

Since it is a much better experience for people that
have flakes enabled. Maybe it could try running flakes
and then fallback to the nix shell?

har7an · 2022-03-02T10:07:32Z

I can do that, of course. Is nix a requirement/hard dependency of nix-shell? I.e. I assume that nix must be present for nix-shell to be present, whereas I can use nix without having nix-shell installed. Is that correct?

when running the dependencies through nix. This makes for a better experience for people having flakes enabled, while still retaining the regular `nix-shell` as fallback for people that don't.

har7an · 2022-03-02T10:24:06Z

@a-kenji Feedback please. :)

Did a quick test, it seems to work fine. I have now built it on top of the following assumptions:

nix-shell cannot exist without nix being present on the host
nix develop ... will only return a nonzero error code if it actually failed to build/launch the nix sandbox (if you can call it that), and will not propagate the error code from the program it executes

If you think that's okay then it's good to go from my POV.

a-kenji · 2022-03-02T10:26:41Z

I can do that, of course. Is nix a requirement/hard dependency of nix-shell? I.e. I assume that nix must be present for nix-shell to be present, whereas I can use nix without having nix-shell installed. Is that correct?

Yes, that is correct. If you have nix installed nix-shell should work, regardless if you use it on a mac, nixos, or a different linux system.

a-kenji · 2022-03-02T10:29:34Z

@a-kenji Feedback please. :)

Did a quick test, it seems to work fine. I have now built it on top of the following assumptions:
* `nix-shell` cannot exist without `nix` being present on the host

* `nix develop ...` will only return a nonzero error code if it actually failed to build/launch the nix sandbox (if you can call it that), and will **not** propagate the error code from the program it executes
If you think that's okay then it's good to go from my POV.

These assumptions seem sensible to me.

a-kenji

LGTM, thank you for all that awesome work and testing!
The documentation can always use helping hands, and If that makes it easier for some people it was well worth all the effort imo!

Containerfile

har7an · 2022-03-02T11:35:47Z

Happy to help, thanks for all the feedback and debugging nix-related stuff! :)

a-kenji force-pushed the main branch from 58bb350 to db85192 Compare February 24, 2022 09:22

har7an force-pushed the feature/containerize branch from dbfa235 to 8f415c9 Compare February 28, 2022 08:56

har7an added 2 commits March 1, 2022 07:59

Containerfile: Add a container for the docs

5f05cb2

that includes the doc dependencies. Builds a container of about 145 MB that builds and serves the docs locally.

watch-serve: Remove containers after use

3e51fd9

Add the `--rm` flag to automatically delete the containers after they have been used. The script won't restart existing containers anyways, so there is no point in keeping them. It will only fill up the users disk.

har7an force-pushed the feature/containerize branch from 8f415c9 to 3e51fd9 Compare March 1, 2022 07:00

watch-serve.sh: Add nix support

62a6457

Calls the script with `nix-shell --command` if it detects that nix is present on the host.

watch-serve.sh: Prefer 'nix develop' over 'nix-shell'

df6dc54

when running the dependencies through nix. This makes for a better experience for people having flakes enabled, while still retaining the regular `nix-shell` as fallback for people that don't.

a-kenji approved these changes Mar 2, 2022

View reviewed changes

Containerfile Show resolved Hide resolved

a-kenji merged commit a8d29c0 into zellij-org:main Mar 2, 2022

har7an deleted the feature/containerize branch March 2, 2022 11:37

Containerize dependencies #92

Containerize dependencies #92

Conversation

har7an commented Feb 24, 2022

a-kenji commented Feb 24, 2022

har7an commented Feb 24, 2022

a-kenji commented Feb 24, 2022

har7an commented Feb 24, 2022

har7an commented Feb 24, 2022

a-kenji commented Feb 24, 2022

a-kenji commented Feb 24, 2022 • edited Loading

har7an commented Feb 24, 2022

har7an commented Feb 24, 2022

a-kenji commented Feb 24, 2022 • edited Loading

har7an commented Feb 25, 2022

a-kenji commented Feb 25, 2022 • edited Loading

har7an commented Feb 25, 2022

a-kenji commented Feb 25, 2022

har7an commented Feb 25, 2022

a-kenji commented Feb 25, 2022

a-kenji commented Feb 25, 2022

har7an commented Feb 25, 2022

a-kenji commented Feb 25, 2022

har7an commented Feb 28, 2022

har7an commented Feb 28, 2022

a-kenji commented Feb 28, 2022 • edited Loading

har7an commented Feb 28, 2022

a-kenji commented Feb 28, 2022

har7an commented Feb 28, 2022

a-kenji commented Feb 28, 2022

har7an commented Mar 1, 2022

har7an commented Mar 1, 2022

a-kenji commented Mar 2, 2022

har7an commented Mar 2, 2022

har7an commented Mar 2, 2022

a-kenji commented Mar 2, 2022

a-kenji commented Mar 2, 2022

a-kenji left a comment

Choose a reason for hiding this comment

har7an commented Mar 2, 2022

a-kenji commented Feb 24, 2022 •

edited

Loading

a-kenji commented Feb 24, 2022 •

edited

Loading

a-kenji commented Feb 25, 2022 •

edited

Loading

a-kenji commented Feb 28, 2022 •

edited

Loading