Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate python packages from nose to pynose #311054

Closed
mweinelt opened this issue May 12, 2024 · 15 comments
Closed

Migrate python packages from nose to pynose #311054

mweinelt opened this issue May 12, 2024 · 15 comments

Comments

@mweinelt
Copy link
Member

mweinelt commented May 12, 2024

The nose test runner will never run on Python 3.12, and we're replacing it with pynose instead. On top of that, packages relying on nose have largely been unmaintained and might be dead leafs or using another test framework entirely. It is also to be expected that they need to be converted to the PEP517 builder in the process of fixing them.

The following packages are in the python3Packages scope and should be addressed first. The final goal is the removal of nose and related packages.

@mweinelt mweinelt changed the title Migrate python packages using nose to pynose Migrate python packages from nose to pynose May 12, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue May 12, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue May 12, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue May 12, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue May 12, 2024
See NixOS#311054, also reenables tests since they work again.
@pyrox0 pyrox0 mentioned this issue May 12, 2024
13 tasks
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue May 25, 2024
gador added a commit to gador/nixpkgs that referenced this issue Jul 8, 2024
fixes build on python3.12
part of NixOS#311054

Signed-off-by: Florian Brandes <florian.brandes@posteo.de>
gador added a commit to gador/nixpkgs that referenced this issue Jul 8, 2024
fixes build on python3.12
part of NixOS#311054

Signed-off-by: Florian Brandes <florian.brandes@posteo.de>
@MostAwesomeDude
Copy link
Contributor

In mdmintz/pynose#16 and mdmintz/pynose#26 it was revealed that pynose is not licenses.mit as announced. I think it should probably inherit the license from nose, which is currently licenses.lgpl3. If you'd like me to fix this, ping me and I'll send a PR referencing this issue.

@eli-schwartz
Copy link

Given that pynose is illegal, copyright-violating software, which by virtue of violating the GPL has lost its right to use the software under the terms of the GPL, it's not remotely clear to me that NixOS is legally allowed to redistribute the tainted edition of the software at all.

I guess I'm not really surprised that some people automatically assume they can do so regardless.

@mweinelt mweinelt closed this as not planned Won't fix, can't repro, duplicate, stale Jul 8, 2024
@mweinelt
Copy link
Member Author

mweinelt commented Jul 8, 2024

I'm working on a treewide removal of pynose.

@SomeoneSerge
Copy link
Contributor

In mdmintz/pynose#16 and mdmintz/pynose#26 it was revealed that pynose is not licenses.mit as announced.

I'm not familiar with nose/pynose, but I'm wondering if something like lib.licenses.unfree // { shortName = "disputed"; } could be adopted for similar situations. Ofc this would nonetheless require making the package a leaf node, possibly rendering the exercise futile

@mweinelt
Copy link
Member Author

mweinelt commented Jul 8, 2024

possibly rendering the exercise futile

That.

@eli-schwartz
Copy link

Nonfree typically means the upstream project legally has the right to be the upstream project, they simply choose to only grant you the ability to use that code under a non OSI approved license.

Where does the concept of a "disputed" license value come from? Under what terms and conditions could or would a user install such a thing, even as a leaf package?

@SomeoneSerge
Copy link
Contributor

Nonfree typically means the upstream project legally has the right to be the upstream project

That sounds reasonable in the context of natural languages. In the context of the metadata that comes with derivations, it's just a single boolean which an individual maintainer could set for whatever reasons... the minimal semantics I could infer from it is "not to be redistributed by Hydra" and "just because you can build it doesn't mean anyone's allowed you to use or distribute it"

Under what terms and conditions could or would a user install such a thing, even as a leaf package?

On terms of taking their own risk; I wouldn't mind shortName = "illegal" either

@mweinelt
Copy link
Member Author

mweinelt commented Jul 8, 2024

The upstream has no right to redistribute the code under the given license. That is the issue we run into.

@eli-schwartz
Copy link

That sounds reasonable in the context of natural languages. In the context of the metadata that comes with derivations, it's just a single boolean which an individual maintainer could set for whatever reasons...

In the context of naturally attempting to convey useful and usable semantic meaning for conventional language concepts, it sounds reasonable to say "if an individual maintainer sets a deceptive value that deceives the user into associating the meaning with that of projects which are under proprietary, but entirely legal, licenses -- the individual maintainer is acting against the best interests of the project and shall receive an infraction by whichever disciplinary measure the distribution in question uses to penalize bad behavior".

I don't quite see the purpose in defending the notion of packaging illegal software using language intended to seem similar to commercial+proprietary redistributables. Is this a thought experiment or a practical desire?

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Jul 9, 2024

Sorry @eli-schwartz, I hadn't read the entirety of 16 and 26 when I first commented, and now, maybe as a result, we're mixing a discussion about implementation details and one about policies.

Is this a thought experiment or a practical desire?

Semi-practical. We've several expressions that describe how to build certain pieces of "legal" software for which, however, we do not have (but the user might) a legal way of obtaining the sources or for which we wouldn't have the right to distribute the results. This is not quite the same as the present situation, but bear with me. These expressions would reference a generally unavailable source, but (a) we wouldn't build them in the CI (ergo never access the source nor distribute), (b) we'd mark them so that they don't evaluate before the user explicitly opts in.

On the implementation side, and I don't particularly like that we confound the names, the latter is practically done by resetting free to false in the "license" description.

nix-repl> runCommand "something" { meta.license = [ { free = false; } ]; } "touch $out"
...
       error: Package ‘something’ in /nix/store/4p0avw1s3vf27hspgqsrqs37gxk4i83i-source/pkgs/build-support/trivial-builders/default.nix:68 has an unfree license (‘unknown’), refusing to evaluate.
...
nix-repl> runCommand "something" { meta.license = [ (lib.licenses.unfree // { shortName = "illegal"; }) ]; } "touch $out"
...
       error: Package ‘something’ in /nix/store/4p0avw1s3vf27hspgqsrqs37gxk4i83i-source/pkgs/build-support/trivial-builders/default.nix:68 has an unfree license (‘illegal’), refusing to evaluate.
...
       Alternatively you can configure a predicate to allow specific packages:
         { nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
             "something"
...

While this error message would be wrong of course (maybe now's a good time to reword it), since there's no, as such, a license granting anything to anyone, the way I read it it does attempt to communicate that "whatever the licensing situation is, you're not automatically permitted to do anything with this software and it's your task to find out the terms and get the consents". Essentially, when I set the license value, I treat free = true as bearing legal meaning (the authors give explicit consent to do FOSS things; in particular, it's sufficient to automatically assume that one may at least build the derivation, but otherwise still has to consult license.meta.url), whereas free = false to me is broader and only means the absence of something.

The licensing information in the pynose expression we have right now is simply wrong. At the very least we'd have to delete mit from the list (resulting in "refusing to evaluate" and "license ('unknown')"), but then we can't depend on it in other packages (lest the CI refuse to build them too). Once these reverse-dependencies are rewritten not to depend on pynose, there'll not be much value in keeping pynose around...

The policy question of when it's worthwhile or acceptable to provide expressions for software which we can't reference a "legally available" source for (proprietary, unlicensed, ...), and whether in particular it's acceptable to reference sources we do know to violate other authors' dissent is global. While this might be worth answering globally "once and for all", we already have, by the previous paragraph, enough local information to remove the package...

EDIT: I seem to have deviated from the original point here. Back to "a thought experiment or a practical desire". Representing correct knowledge about upstream projects definitely is a goal. The present issue is an edge case where our tooling (meta.license) can't meaningfully represent our knowledge, and it's a reminder that our tooling hasn't been entirely adequate for modeling points in the interior of the spectrum (like the nvidia stuff), and it wouldn't have been adequate for representing e.g. unlicensed projects either. So, "semi-practical"

@eli-schwartz
Copy link

Interesting implementation details!

In gentoo, there is a global repository directory containing license terms and conditions, and the file profiles/license_groups maps many registered license to some meta-group such as (random sampling):

  • @FREE
  • @GPL-COMPATIBLE
  • @FSF-APPROVED
  • @OSI-APPROVED-FREE
  • @MISC-FREE
  • @BINARY-REDISTRIBUTABLE
  • @EULA

Users would have to specify which licenses they accept to use (the default is ACCEPT_LICENSE="-* @FREE"). Most people probably do not add ACCEPT_LICENSE="JetBrains-business" or some such, so the result is if you try to install certain packages...

!!! All ebuilds that could satisfy "goland" have been masked.
!!! One of the following masked packages is required to complete your request:
- dev-util/goland-2022.3.3-r1::gentoo (masked by: || ( JetBrains-business JetBrains-classroom JetBrains-educational JetBrains-individual ) license(s), ~amd64 keyword)
A copy of the 'JetBrains-business' license is located at '/var/db/repos/gentoo/licenses/JetBrains-business'.

A copy of the 'JetBrains-classroom' license is located at '/var/db/repos/gentoo/licenses/JetBrains-classroom'.

A copy of the 'JetBrains-educational' license is located at '/var/db/repos/gentoo/licenses/JetBrains-educational'.

A copy of the 'JetBrains-individual' license is located at '/var/db/repos/gentoo/licenses/JetBrains-individual'.


For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

Project policy is to "ask the opinion of others before committing a new license, if you're not sure about the license". This is somewhat flexible obviously. The addition of new licenses is, incidentally, easy to coordinate and watch for changes. :D Getting the license wrong is a very human mistake, adding something that isn't a license at all would probably raise a few eyebrows.

There's no real room to mark a package as nonfree without stating the specific reason it isn't free, since "unknown" licenses are banned by CI-enforced policy.

The license groups in particular are very neat. It encourages accurate specification of "terms and conditions" independent of "is this license one that I consider free", since you can have arbitrarily grouped or recursively-grouped license sets to describe policy with, instead.

@eli-schwartz
Copy link

Ah, w.r.t. the edit: one could ACCEPT_LICENSE="NVIDIA-CUDA", or since that is part of the @EULA group, ACCEPT_LICENSE="@EULA" if one doesn't mind an EULA in general.

@SomeoneSerge
Copy link
Contributor

SomeoneSerge commented Jul 9, 2024

Nixpkgs is orderly enough about "free and opensource" licenses (there's a list maintained in a single place which other expressions reference, e.g. license = [ lib.licenses.mit ]), but the rest can be fairly ad hoc at times. We'll surely look at Gentoo for inspiration when we try to bring some order to this part of Nixpkgs, or when we discuss the policy

As for NVidia, one approach usable today could look something like this:

allowUnfreePredicate =
p:
builtins.all (
license:
license.free
|| license.redistributable
|| builtins.elem license.shortName [
"CUDA EULA"
"cuDNN EULA"
"NVidia OptiX EULA"
]
) (ensureList p.meta.license);

pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue Jul 9, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue Jul 9, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue Jul 9, 2024
pyrox0 added a commit to pyrox0/nixpkgs that referenced this issue Jul 9, 2024
See NixOS#311054, also reenables tests since they work again.
Shawn8901 pushed a commit to Shawn8901/nixpkgs that referenced this issue Jul 9, 2024
part of NixOS#311054

Signed-off-by: Florian Brandes <florian.brandes@posteo.de>
erictapen pushed a commit to erictapen/nixpkgs that referenced this issue Jul 9, 2024
part of NixOS#311054

Signed-off-by: Florian Brandes <florian.brandes@posteo.de>
github-actions bot pushed a commit that referenced this issue Jul 9, 2024
part of #311054

Signed-off-by: Florian Brandes <florian.brandes@posteo.de>
(cherry picked from commit 30a05b5)
@emilazy
Copy link
Member

emilazy commented Jul 12, 2024

Spiritual successor: #326513

@mdmintz
Copy link

mdmintz commented Jul 12, 2024

This should hopefully help clear up any confusion: mdmintz/pynose#36

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants