Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildLuaPackage: enable __structuredAttrs rocks #224553

Merged
merged 4 commits into from
Apr 23, 2023

Conversation

teto
Copy link
Member

@teto teto commented Apr 4, 2023

it makes overriding easier, instead of having to know internals to decide which of sqlite = prev.luaLib.overrideLuarocks prev.sqlite (drv: { or sqlite = prev.sqlite.overrideAttrs (drv: { just use the latter

Description of changes
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@teto
Copy link
Member Author

teto commented Apr 9, 2023

cc @Artturin @ncfavier as structured attrs experts could you check/advise if I am doing anything wrong here ? I feel like structuredAttrs could solve many lua related issues and I would like to get this in before the 23.05 release.

it makes overriding easier, instead of having to know internals to
decide which of `sqlite = prev.luaLib.overrideLuarocks prev.sqlite (drv: {` or
`sqlite = prev.sqlite.overrideAttrs (drv: {` just use the latter
@teto teto marked this pull request as ready for review April 23, 2023 00:28
@teto
Copy link
Member Author

teto commented Apr 23, 2023

nixpkgs-review gives 2 errors for packages that are already broken on master (nvimpager and a lua library that doesn't support lua.4) I tink that's good to merge @vcunat @figsoda

@teto teto merged commit 70c68b1 into NixOS:master Apr 23, 2023
@teto teto deleted the lua-structured-attrs branch April 23, 2023 18:45
@arcnmx
Copy link
Member

arcnmx commented Apr 24, 2023

This seems to have broken a lot in buildLuarocksPackage?

  1. if a meta attr isn't passed, it now fails evaluation with error: attribute 'meta' missing
  2. passthru from attrs is now ignored/overwritten if passed.
  3. basically any attrs assigned in the builder now overwrite any passed via attrs inputs (including buildInputs)

I started trying to fix/revert some of this but I kept discovering more problems and gave up. I think this needs more attention before merging?

@teto
Copy link
Member Author

teto commented Apr 24, 2023

do you have specific examples in mind ? maybe you hit this because you are running out of tree lua modules ? I ran nixpkgs-review on it and was able to build everything. There should be no need to pass a passthru anymore. The meta might be broken indeed.

@arcnmx
Copy link
Member

arcnmx commented Apr 24, 2023

maybe you hit this because you are running out of tree lua modules

Well yes, this applies to anyone who might call or use these functions, now they rigidly only work specifically for the packages tested with. I'm under the impression that there's precedence (and expectation) that builders won't just ignore and blindly override input attrs passed by the package using them - all the code that was in place to prevent that from happening has been removed by this PR.

There should be no need to pass a passthru anymore

passthru is just one of many attrs that a package may want to pass to mkDerivation for countless reasons - as a builder is a wrapper around mkDerivation, one expects it to be respected if passed.

@teto
Copy link
Member Author

teto commented Apr 24, 2023

do you have some public reference of outside usage . Since so few people contribute to the lua ecosystem I am under the (apparently wrong) impression that no one uses it.

I think this needs more attention before merging?

This PR changed a lot of things with the goal to make it easier to use: this removes the need to know when to call overrideLuarocks and overrideAttrs (tip: just use the latter). It took a lot of work and time to get there, if only because building those packages need CPU and disk storage. I am sorry if this broke your workflow but the fixes are trivial in comparison, especially now that the packages are cached. Do not hesitate to add tests to avoid this kind of regression. We have the distinction between unstable and stable precisely for this reason. And nixpkgs makes it easy to rollback.

@Artturin
Copy link
Member

@teto passthru is very important even with structuredAttrs to avoid unnecessary attrs becoming a part of the derivation (causing unnecessary rebuilds)

@teto
Copy link
Member Author

teto commented Apr 24, 2023

yes but it can still be used, just differently until I fix the call (and it is not used within nixpkgs)

broken = disabled;
} // meta;
} // attrs.meta;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attrs.meta or {}

inherit externalDeps;
inherit luarocks_content;
} // passthru;
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

} // attrs.passthru or {};

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also env is important to preserve too

@@ -167,7 +167,7 @@ let
wrapLuaPrograms
'' + attrs.postFixup or "";

installPhase = attrs.installPhase or ''
Copy link
Member

@arcnmx arcnmx Apr 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sorry if this broke your workflow but the fixes are trivial in comparison

I understand it's an impactful/breaking change to the builder in general, the only part I really take issue with here is that no care or attention was given to preserving input attrs.

The lines with or/++/// like this one (and all the others: meta, passthru, buildInputs, installPhase, checkPhase, etc) were removed while they could've just been left as-is - this leaves the function incorrect and less flexible than it was before, for no benefit or change. It's just silly to remove lines unrelated to your changes that serve a purpose 😞

teto added a commit to teto/nixpkgs that referenced this pull request Apr 24, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit to teto/nixpkgs that referenced this pull request Apr 24, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit to teto/nixpkgs that referenced this pull request Apr 25, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit to teto/nixpkgs that referenced this pull request Apr 27, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit to teto/nixpkgs that referenced this pull request Apr 27, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit to teto/nixpkgs that referenced this pull request Apr 27, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit that referenced this pull request Apr 27, 2023
follow up of #224553 where
some arguments got ignored whil they were before taken into account.
teto added a commit to teto/nixpkgs that referenced this pull request Apr 27, 2023
follow up of NixOS#224553 where
some arguments got ignored whil they were before taken into account.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants