Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixopsUnstable broke after upgrade recently #139532

Closed
aij opened this issue Sep 26, 2021 · 16 comments
Closed

nixopsUnstable broke after upgrade recently #139532

aij opened this issue Sep 26, 2021 · 16 comments
Labels
0.kind: bug Something is broken 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md

Comments

@aij
Copy link
Contributor

aij commented Sep 26, 2021

Describe the bug

Nixops was working before upgrading

Steps To Reproduce

Steps to reproduce the behavior:

  1. Use nixops
  2. Upgrade nixpkgs to any version later than 45256745cef246dabe1ae8a7d109988f190cd7ef
  3. Run nixops deploy or even nixops list

Expected behavior

I should be able to run nixops deploy to update a deployment.

Screenshots

ivan@tobati:~/nixops/aij$ nixops deploy --build-only
error: while evaluating 'zipAttrs' at /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:17:14, called from /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:38:13:
while evaluating 'operator' at /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:31:20, called from undefined position:
while evaluating 'getNetworkFromExpr' at /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:24:28, called from /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:31:45:
while evaluating 'call' at /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:12:10, called from /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:25:10:
anonymous function at /home/ivan/nixops/aij/nixops.nix:1:1 called without required argument 'config', at /nix/store/flaida1s0xg1f6wrybq17834nbbyhkk8-python3-3.9.6-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:12:43
error: evaluation of the deployment specification failed

Additional context

I re-tested with previous version of nixos-unstable-small I had used, and found this changed between b769bd2b01bce5afe0af703218593355c6f52337 and f96ba73ac0057f5da7920b9635b25938508ad39b. Looking at git long and a bit more testing I found the 942b5b6 included a nixopsUnstable upgrade which seems like the cause.

I have not (knowingly) started using flakes yet.

I'm not entirely sure this is a proper bug or an intentional backwards incompatibility with an obtuse error message. If nothing else we should improve the error message and document in release notes.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

ivan@tobati:~/nixops/aij$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.63, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - channels(root): `"nixos-20.03pre208171.2e8fc97dbfa"`
 - nixpkgs: `/etc/nixos/aij/unstable`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
@aij aij added the 0.kind: bug Something is broken label Sep 26, 2021
@Izorkin
Copy link
Contributor

Izorkin commented Sep 26, 2021

@aij see NixOS/nixops#1474
Latest nixopsUnstable working only nix 2.4 and not correct working with multiple deployments :(

@aij
Copy link
Contributor Author

aij commented Sep 26, 2021

https://nixops.readthedocs.io/en/latest/manual/migrating.html is rather unclear about what needs to go in the new ./nixops.nix besides network.storage.legacy = {};

From what I gather, it needs to include the entire deployment configuration, but even then I'm seeing some weird behavior, like nixops ssh-for-each not doing anything and

ivan@tobati:~/nixops$ nixops check
Machines state:
+------+--------+----+-----------+----------+-----------+-------+-------+
| Name | Exists | Up | Reachable | Disks OK | Load avg. | Units | Notes |
+------+--------+----+-----------+----------+-----------+-------+-------+
+------+--------+----+-----------+----------+-----------+-------+-------+
Non machines resources state:
+------+--------+
| Name | Exists |
+------+--------+
+------+--------+

I'm not sure what I did that was relevant, (perhaps nixops deploy --build-only) but it seems to have gotten into a better state now:

ivan@tobati:~/nixops$ nixops ssh-for-each uptime
m0...>  14:35:11  up 27 days 20:22,  0 users,  load average: 0.39, 0.37, 0.41
m3...>  14:35:11  up  22:15,  4 users,  load average: 0.81, 0.84, 0.76
m4...>  14:35:12  up 27 days 20:13,  0 users,  load average: 1.52, 1.20, 0.84
m5...>  14:35:13  up 33 days 18:13,  0 users,  load average: 0.05, 0.09, 0.09
m6...>  14:35:13  up 31 days 19:15,  0 users,  load average: 0.30, 0.36, 0.44
ivan@tobati:~/nixops$ nixops check
Machines state:
+------+--------+-----+-----------+----------+----------------+---------------------------+-------+
| Name | Exists | Up  | Reachable | Disks OK | Load avg.      | Units                     | Notes |
+------+--------+-----+-----------+----------+----------------+---------------------------+-------+
| m0   | Yes    | Yes | Yes       | N/A      | 1.57 0.89 0.60 |                           |       |
| m3   | Yes    | Yes | Yes       | N/A      | 0.44 0.62 0.69 | dev-fuse.device [running] |       |
| m4   | Yes    | Yes | Yes       | N/A      | 0.55 0.87 0.78 |                           |       |
| m5   | Yes    | Yes | Yes       | N/A      | 0.00 0.04 0.07 | m3-scratch.mount [failed] |       |
|      |        |     |           |          |                | scratch.mount [failed]    |       |
| m6   | Yes    | Yes | Yes       | N/A      | 0.39 0.36 0.43 | m3-scratch.mount [failed] |       |
|      |        |     |           |          |                | m4-scratch.mount [failed] |       |
+------+--------+-----+-----------+----------+----------------+---------------------------+-------+
Non machines resources state:
+------+--------+
| Name | Exists |
+------+--------+
+------+--------+

@Izorkin
Copy link
Contributor

Izorkin commented Sep 26, 2021

@aij check this patch:

diff --git a/pkgs/applications/networking/cluster/nixops/default.nix b/pkgs/applications/networking/cluster/nixops/default.nix
index 91f5cb82668..6515885fd8e 100644
--- a/pkgs/applications/networking/cluster/nixops/default.nix
+++ b/pkgs/applications/networking/cluster/nixops/default.nix
@@ -17,10 +17,6 @@ let

             nixops = super.nixops.overridePythonAttrs (
               old: {
-                postPatch = ''
-                  substituteInPlace nixops/args.py --subst-var version
-                '';
-
                 meta = old.meta // {
                   homepage = https://github.com/NixOS/nixops;
                   description = "NixOS cloud provisioning and deployment tool";
diff --git a/pkgs/applications/networking/cluster/nixops/poetry-git-overlay.nix b/pkgs/applications/networking/cluster/nixops/poetry-git-overlay.nix
index cc40395d6d9..3f0bbece83a 100644
--- a/pkgs/applications/networking/cluster/nixops/poetry-git-overlay.nix
+++ b/pkgs/applications/networking/cluster/nixops/poetry-git-overlay.nix
@@ -5,8 +5,8 @@ self: super: {
     _: {
       src = pkgs.fetchgit {
         url = "https://github.com/NixOS/nixops.git";
-        rev = "35ac02085169bc2372834d6be6cf4c1bdf820d09";
-        sha256 = "1jh0jrxyywjqhac2dvpj7r7isjv68ynbg7g6f6rj55raxcqc7r3j";
+        rev = "30e16880e7afc08ca938266e3a3b3e0c6464682a";
+        sha256 = "sha256-ALJrw0qMMQb0KeU2CgVsByiRIwvoegVA0wpV0ZBjF/w=";
       };
     }
   );

Revert nixops to version 2.0.0-pre (2021-06-15).

@roberth
Copy link
Member

roberth commented Oct 17, 2021

@Izorkin

Latest nixopsUnstable working only nix 2.4 and not correct working with multiple deployments :(

#142035 has a working test case for non-flake use.

@Izorkin
Copy link
Contributor

Izorkin commented Oct 23, 2021

@roberth this method use single deployments. I need support for multiple deployments.

@stale
Copy link

stale bot commented Apr 25, 2022

I marked this as stale due to inactivity. → More info

@Izorkin
Copy link
Contributor

Izorkin commented Apr 25, 2022

Problem not resolved.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 25, 2022
@Izorkin
Copy link
Contributor

Izorkin commented Jun 16, 2022

@roberth there is some progress?

@roberth
Copy link
Member

roberth commented Jul 5, 2022

This was fixed recently and has landed in nixos-unstable-small.

@roberth roberth closed this as completed Jul 5, 2022
@Izorkin
Copy link
Contributor

Izorkin commented Jul 5, 2022

@roberth error running command nixops info --all from home directory:

Traceback (most recent call last):
  File "/nix/store/cs9brx0b0nwbvr0ca11jhswxs8xzhh56-python3.9-nixops-2.0.0-pre-7220cbd/bin/.nixops-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nixops/__main__.py", line 56, in main
    args.op(args)
  File "/nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nixops/script_defs.py", line 426, in op_info
    with network_state(args, False, "nixops info") as sf:
  File "/nix/store/as5aamjfk9da5napyp5hpn6hha6f7cmp-python3-3.9.13/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nixops/script_defs.py", line 102, in network_state
    network = eval_network(get_network_file(args))
  File "/nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nixops/script_defs.py", line 60, in get_network_file
    raise ValueError(f"Neither flake.nix nor nixops.nix exists in {network_dir}")
ValueError: Neither flake.nix nor nixops.nix exists in /home/rebrain

from nixpkgs directory:

error: writing to file: No space left on device

       … while fetching the input 'path:/home/rebrain/nixpkgs'

       … while evaluating 'call'

       at /nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:12:10:

           11| let
           12|   call = x: if builtins.isFunction x then x args else x;
             |          ^
           13|

       … from call site

       at /nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:36:11:

           35|       ++ optional (flakeUri != null)
           36|         ((call flakeExpr) // { _file = "<${flakeUri}>"; });
             |           ^
           37|

       … while evaluating 'zipAttrs'

       at /nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:17:14:

           16|
           17|   zipAttrs = set: builtins.listToAttrs (
             |              ^
           18|     map (name: { inherit name; value = builtins.catAttrs name set; }) (builtins.concatMap builtins.attrNames set));

       … from call site

       at /nix/store/h7x6j5ap29nnrf09yf39n8dr0vqj9k19-python3-3.9.13-env/lib/python3.9/site-packages/nix/eval-machine-info.nix:38:13:

           37|
           38|   network = zipAttrs networks;
             |             ^
           39|

Version 1.7 starts from anywhere normally:

/nix/store/xfnix5aknqqlhqr64wbmmy4hvzmyzd0k-python2.7-apache-libcloud-2.8.3/lib/python2.7/site-packages/libcloud/common/google.py:93: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.backends import default_backend
trace: warning: The extraArgs argument to eval-config.nix is deprecated. Please set config._module.args instead.
trace: warning: The extraArgs argument to eval-config.nix is deprecated. Please set config._module.args instead.
trace: warning: The extraArgs argument to eval-config.nix is deprecated. Please set config._module.args instead.
trace: warning: The extraArgs argument to eval-config.nix is deprecated. Please set config._module.args instead.
+------------+------------+-----------------+------+--------------------------------------------------------+------------+
| Deployment | Name       |      Status     | Type | Resource Id                                            | IP address |
+------------+------------+-----------------+------+--------------------------------------------------------+------------+
| work       | vm01-web   | Up / Up-to-date | none | nixops-4bef9dae-734f-11eb-94ab-52cff3c67bdb-vm01-web   |            |
| work       | vm02-mail  | Up / Up-to-date | none | nixops-4bef9dae-734f-11eb-94ab-52cff3c67bdb-vm02-mail  |            |
| work       | vm03-tools | Up / Up-to-date | none | nixops-4bef9dae-734f-11eb-94ab-52cff3c67bdb-vm03-tools |            |
| test       | vm11-tech  | Up / Up-to-date | none | nixops-500ff938-734f-11eb-a56c-52cff3c67bdb-vm11-tech  |            |
+------------+------------+-----------------+------+--------------------------------------------------------+------------+

@Izorkin
Copy link
Contributor

Izorkin commented Jul 5, 2022

Create folder ~/test and write file ~/test/nixops.nix:

{
  network.storage.legacy = {
    databasefile = "~/.nixops/deployments.nixops";
  };
}

Running nixops deploy -d test:

resource ‘vm11-tech’ is obsolete
building all machine configurations...
this derivation will be built:
  /nix/store/kcmz6i31sf8lhnkfxywnchwbf0clq33c-nixops-machines.drv
building '/nix/store/kcmz6i31sf8lhnkfxywnchwbf0clq33c-nixops-machines.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
labs> closures copied successfully
labs> deployment finished successfully

@roberth
Copy link
Member

roberth commented Jul 6, 2022

@roberth error running command nixops info --all from home directory:

Commands that should list the legacy deployments are known to have regressed, as NixOps2 has configurable state backends that require it to start by evaluating the network.

error: writing to file: No space left on device

   … while fetching the input 'path:/home/rebrain/nixpkgs'

It's trying to evaluate the working directory as a flake. It should try git+file first. NixOps doesn't seem to add path:, so this specific error seems to be a nix problem; one that will be resolved by NixOS/nix#6530 regardless of "uri" scheme.

@Izorkin
Copy link
Contributor

Izorkin commented Jul 6, 2022

Is it possible to make a config file ~/.config/nixops/default.nix so that specify 'network.storage.legacy' parameter there and keep legacy deployments. And also save the launch from any folder?

@roberth
Copy link
Member

roberth commented Jul 6, 2022

Such functionality would have to take priority over the working directory and therefore break "gitops" style deployments.
Even if it is "optional" because you'd have to create the dotfile, it would still break when you try to use a gitops style deployment later. The resulting behavior would be confusing or perhaps even dangerous when scripts are involved. You won't be able to switch between old and new style deployments easily because of the dotfile.

If I may generalize, such global and implicit behavior is confusing and atypical for an ecosystem that promotes hermeticity, reproducibility, etc. NixOps shouldn't have behaved like this in the first place, but it is a tool that was first created in a different era.

@Izorkin
Copy link
Contributor

Izorkin commented Jul 6, 2022

@roberth how to create a new deployment in nixops 2.0 so that it can be run from the nixpkgs folder? We need this variant to make it easier to test changes to nixpkgs on a remote virtual machine.
Is it possible to make two deployments in one folder?

@Izorkin
Copy link
Contributor

Izorkin commented Nov 8, 2023

@roberth how do I get the old nixopsUnstable behavior back? NixopsStable has now been removed :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
Projects
None yet
Development

No branches or pull requests

3 participants