-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build Docker images with Nix #36
base: main
Are you sure you want to change the base?
Changes from 2 commits
3c9bc56
b68716c
2867d33
b20aa90
47a8fe7
5c44ae3
fee2bae
25b219d
4fc323b
ed8f9ef
2cf882d
e9db7d5
a837243
f7be978
263f076
c498a36
c243e00
4def9cf
3b1e15e
788675e
d0fe09a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
|
||
--- | ||
name: Docker Nix Images | ||
|
||
on: | ||
push: | ||
branches: | ||
- main | ||
paths: | ||
- '.github/workflows/docker-nix.yml' | ||
- 'nixpkgs.json' | ||
- '*.nix' | ||
- '*/*.nix' | ||
pull_request: | ||
branches: | ||
- main | ||
paths: | ||
- '.github/workflows/docker-nix.yml' | ||
- 'nixpkgs.json' | ||
- '*.nix' | ||
- '*/*.nix' | ||
|
||
jobs: | ||
build-nix: | ||
name: Build Docker images with Nix | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v3 | ||
|
||
- name: Install Nix | ||
uses: cachix/install-nix-action@v20 | ||
with: | ||
nix_path: nixpkgs=channel:nixos-22.11 | ||
|
||
- name: Build the Docker images with nix | ||
run: | | ||
nix-build | while read IMG; do | ||
docker load --input ${IMG} | ||
done | ||
|
||
- name: List images | ||
run: docker images |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ pkgs ? import ./nixpkgs.nix { } }: | ||
{ | ||
mailbox = pkgs.callPackage ./mailbox/docker-image.nix { }; | ||
relay = pkgs.callPackage ./relay/docker-image.nix { }; | ||
wormhole = pkgs.callPackage ./wormhole/docker-image.nix { }; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
{ dockerTools, python3, lib }: | ||
let | ||
pname = "magic-wormhole-mailbox-server"; | ||
# Inject mailbox in the standard Python env | ||
pyenv = python3.buildEnv.override { | ||
extraLibs = [ python3.pkgs.${pname} ]; | ||
ignoreCollisions = true; | ||
}; | ||
ver = { | ||
py = lib.concatStringsSep "" ( lib.lists.sublist 0 2 ( lib.strings.splitString "." python3.version ) ); | ||
pkg = python3.pkgs.${pname}.version; | ||
nix = lib.concatStringsSep "" ( lib.lists.sublist 0 2 ( lib.strings.splitString "." lib.version ) ); | ||
}; | ||
in | ||
# Build the image with our custom CMD | ||
dockerTools.buildLayeredImage { | ||
name = pname; | ||
tag = "${ver.pkg}-python${ver.py}-nix${ver.nix}"; | ||
config = { | ||
WorkingDir = "/app"; | ||
Volumes = { "/db" = { }; }; | ||
Cmd = [ | ||
"twist wormhole-mailbox" | ||
btlogy marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"--usage-db=/db/usage-relay.sqlite" | ||
"--blur-usage=3600" | ||
"--channel-db=/db/relay.sqlite" | ||
]; | ||
}; | ||
contents = [ pyenv ]; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
{ | ||
"name": "source", | ||
"url": "https://releases.nixos.org/nixos/21.11/nixos-21.11.337975.eabc3821918/nixexprs.tar.xz", | ||
"sha256": "1fq3zz7qfavksdbqvicns7hg61q3hhbxs2ibm818gy629hwkvsvm" | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Import local nixpkg.json which pins all our Nix packages | ||
import (builtins.fetchTarball (builtins.fromJSON (builtins.readFile ./nixpkgs.json))) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Might want to consider using a tool like npins or niv here, it will also make updating the pin much more pleasant There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was looking into niv, but I could not see how/if it is used for PrivateStorageio! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @exarkun : how do we update nixpkgs in PrivateStorageio? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The shell script The python script already broke once because the HTTP library it uses changed its API and we didn't have its version pinned. That's why the moment @exarkun turns his back I'll replace the whole thing by my 12 line There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thank you @hacklschorsch. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. FWIW I think there are nix commands built-in when using flakes … There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did not dare pushing this experiment to the flake level :-) I'll see later if I can spend some extra time to improve this (with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I won't be there this week because national holiday, but next week I could talk a bit about these There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No business value in adding yet another way to update one three line json file. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
{ dockerTools, python3, lib }: | ||
let | ||
pname = "magic-wormhole-transit-relay"; | ||
# Inject mailbox in the standard Python env | ||
pyenv = python3.buildEnv.override { | ||
extraLibs = [ python3.pkgs.${pname} ]; | ||
ignoreCollisions = true; | ||
}; | ||
ver = { | ||
py = lib.concatStringsSep "" ( lib.lists.sublist 0 2 ( lib.strings.splitString "." python3.version ) ); | ||
pkg = python3.pkgs.${pname}.version; | ||
nix = lib.concatStringsSep "" ( lib.lists.sublist 0 2 ( lib.strings.splitString "." lib.version ) ); | ||
}; | ||
in | ||
# Build the image with our custom CMD | ||
dockerTools.buildLayeredImage { | ||
name = pname; | ||
tag = "${ver.pkg}-python${ver.py}-nix${ver.nix}"; | ||
config = { | ||
WorkingDir = "/app"; | ||
Volumes = { "/db" = { }; }; | ||
Cmd = [ | ||
"twist transitrelay" | ||
btlogy marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"--usage-db=/db/usage-transitrelay.sqlite" | ||
"--blur-usage=3600" | ||
"--port=tcp:4001" | ||
"--websocket=tcp:4002" | ||
]; | ||
}; | ||
contents = [ pyenv ]; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
{ dockerTools, python3, lib }: | ||
let | ||
pname = "magic-wormhole"; | ||
# Inject mailbox in the standard Python env | ||
pyenv = python3.buildEnv.override { | ||
extraLibs = [ python3.pkgs.${pname} ]; | ||
ignoreCollisions = true; | ||
}; | ||
ver = { | ||
py = lib.concatStringsSep "" ( lib.lists.sublist 0 2 ( lib.strings.splitString "." python3.version ) ); | ||
pkg = python3.pkgs.${pname}.version; | ||
nix = lib.concatStringsSep "" ( lib.lists.sublist 0 2 ( lib.strings.splitString "." lib.version ) ); | ||
}; | ||
in | ||
# Build the image with our custom CMD | ||
dockerTools.buildLayeredImage { | ||
name = pname; | ||
tag = "${ver.pkg}-python${ver.py}-nix${ver.nix}"; | ||
config = { | ||
WorkingDir = "/app"; | ||
EntryPoint = [ | ||
"wormhole" | ||
]; | ||
}; | ||
contents = [ pyenv ]; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you wanted this to use PyPy instead, you would use the name of the PyPy derivation here instead - eg "pypy3".
If you want to try this, I suggest using a very recent version of nixpkgs master@HEAD (or waiting for the 23.05 release). I don't know if wormhole will have issues or not but I wasn't able to get Tahoe-LAFS to work on PyPy without a lot of hacking, even on the NixOS 22.11 release branch, due to a lot of Python package incompatibilities with PyPy that hadn't yet been addressed. But after 22.11 it looks like someone has paid a bit closer attention to PyPy support and it's a little easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the tip.
If we want to compare Debian vs NixOS images, PyPy would help indeed.
But I'm not sure how much time it will take (time was also an important factor).