Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib: Create a generic property testing framework #212858

Closed
wants to merge 1 commit into from

Conversation

infinisil
Copy link
Member

Description of changes

In #205190 some property tests were added, but extending those turns out to be pretty hard, which was noticed in #209099 when trying to add more.

This is a draft for a new generic property-testing framework that's pretty simple to use. It should eventually be usable for all the path library functions from #210426

This work is sponsored by Antithesis

Example

Here's a messed up regex for the lib.path.subpath.isValid check:

diff --git a/lib/path/default.nix b/lib/path/default.nix
index 96a9244407b..8c6fa15af44 100644
--- a/lib/path/default.nix
+++ b/lib/path/default.nix
@@ -34,7 +34,7 @@ let
     else if substring 0 1 value == "/" then
       "The given string \"${value}\" starts with a `/`, representing an absolute path"
     # We don't support ".." components, see ./path.md#parent-directory
-    else if match "(.*/)?\\.\\.(/.*)?" value != null then
+    else if match "(.*/)?\\.\\.(/.+)?" value != null then
       "The given string \"${value}\" contains a `..` component, which is not allowed in subpaths"
     else null;

And this is what you get when you try to run the property tests:

$ lib/path/tests/run.sh
Running property test normalise-append
Running property test normalise-error
Property test normalise-error failed: Expected variables valid and normalise_success to be the same, but they're not
[awk] subpath = gen_subpath() =
  ../
[awk] valid = subpath_is_valid(subpath) ? "true" : "false" =
  false
[nix] normalise_success = lib.boolToString (builtins.tryEval (lib.path.subpath.normalise subpath)).success =
  true
To reproduce run: lib/path/tests/run.sh normalise-error 15645

@infinisil
Copy link
Member Author

While this was interesting to create, and awk is a surprisingly good language (except the lack of local variables), the proper way to do this is to have stable language bindings for e.g. Python. For this there's already https://github.com/Mic92/pythonix, but that was archived due to requiring too much maintenance (since it relied on the unstable Nix library interface).

I updated pythonix to the latest Nix version in this fork's commit, but I'm considering upstreaming those bindings to Nix, such that they don't need to be maintained independently anymore.

@infinisil infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023
@roberth roberth added the 6.topic: lib The Nixpkgs function library label Jul 8, 2023
@infinisil
Copy link
Member Author

I'm closing this since there's now a PR to have Python bindings for Nix (outdated, won't get merged), and another one for C bindings (could get merged). Any of those approaches would be better than this hack here :).

@infinisil infinisil closed this Nov 9, 2023
@infinisil infinisil deleted the lib-property-tests branch November 9, 2023 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: lib The Nixpkgs function library significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants