Rework Preferences loading framework

Implements the `Preferences` loading framework as outlined in [0]. The
most drastic change is that the list of compile-time preferences is no
longer sequestered within its own dictionary, but is instead
autodetected at compile-time and communicated back to the compiler.
This list of compile-time preferences is now embedded as an array of
strings that the loader must load, then index into the preferences
dictionary with that list to check the preferences hash.

[0]: JuliaLang/julia#37791 (comment)

Project.toml

@@ -7,11 +7,10 @@ version = "1.0.0"
 TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 
 [extras]
-Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "Pkg"]
+test = ["Test"]
 
 [compat]
 julia = "1.6"

README.md

@@ -1,37 +1,66 @@
 # Preferences
 
-`Preferences` supports embedding a simple `Dict` of metadata for a package on a per-project basis.
-These preferences allow for packages to set simple, persistent pieces of data, and optionally trigger recompilation of the package when the preferences change, to allow for customization of package behavior at compile-time.
+The `Preferences` package provides a convenient, integrated way for packages to store configuration switches to persistent TOML files, and use those pieces of information at both run time and compile time.
+This enables the user to modify the behavior of a package, and have that choice reflected in everything from run time algorithm choice to code generation at compile time.
+Preferences are stored as TOML dictionaries and are, by default, stored within a `(Julia)LocalPreferences.toml` file next to the currently-active project.
+If a preference is "exported", it is instead stored within the `(Julia)Project.toml` instead.
+The intention is to allow shared projects to contain shared preferences, while allowing for users themselves to override those preferences with their own settings in the `LocalPreferences.toml` file, which should be `.gitignore`d as the name implies.
 
-## API Overview
+Preferences can be set with depot-wide defaults; if package `Foo` is installed within your global environment and it has preferences set, these preferences will apply as long as your global environment is part of your [`LOAD_PATH`](https://docs.julialang.org/en/v1/manual/code-loading/#Environment-stacks).
+Preferences in environments higher up in the environment stack get overridden by the more proximal entries in the load path, ending with the currently active project.
+This allows depot-wide preference defaults to exist, with active projects able to merge or even completely overwrite these inherited preferences.
+See the docstring for `set_preferences!()` for the full details of how to set preferences to allow or disallow merging.
 
-`Preferences` are used primarily through the `@load_preferences`, `@save_preferences` and `@modify_preferences` macros.
-These macros will auto-detect the UUID of the calling package, throwing an error if the calling module does not belong to a package.
-The function forms can be used to load, save or modify preferences belonging to another package.
+Preferences that are accessed during compilation are automatically marked as compile-time preferences, and any change recorded to these preferences will cause the Julia compiler to recompile any cached precompilation `.ji` files for that module.
+This allows preferences to be used to influence code generation.
+When your package sets a compile-time preference, it is usually best to suggest to the user that they should restart Julia, to allow recompilation to occur.
 
-Example usage:
+## API
+
+Preferences use is very simple; it is all based around two functions (which each have convenience macros): `@set_preferences!()` and `@load_preference()`.
+
+* `@load_preference(key, default = nothing)`: This loads a preference named `key` for the current package.  If no such preference is found, it returns `default`.
+
+* `@set_preferences!(pairs...)`: This allows setting multiple preferences at once as pairs.
+
+To illustrate the usage, we show a toy module, taken directly from this package's tests:
 
 ```julia
-using Preferences
+module UsesPreferences
+
+function set_backend(new_backend::String)
+    if !(new_backend in ("OpenCL", "CUDA", "jlFPGA"))
+        throw(ArgumentError("Invalid backend: \"$(new_backend)\""))
+    end
 
-function get_preferred_backend()
-    prefs = @load_preferences()
-    return get(prefs, "backend", "native")
+    # Set it in our runtime values, as well as saving it to disk
+    @set_preferences!("backend" => new_backend)
+    @info("New backend set; restart your Julia session for this change to take effect!")
 end
 
-function set_backend(new_backend)
-    @modify_preferences!() do prefs
-        prefs["backend"] = new_backend
+const backend = @load_preference("backend", "OpenCL")
+
+# An example that helps us to prove that things are happening at compile-time
+function do_computation()
+    @static if backend == "OpenCL"
+        return "OpenCL is the best!"
+    elseif backend == "CUDA"
+        return "CUDA; so fast, so fresh!"
+    elseif backend == "jlFPGA"
+        return "The Future is Now, jlFPGA online!"
+    else
+        return nothing
     end
 end
-```
 
-Preferences are stored within the first `Project.toml` that represents an environment that contains the given UUID, even as a transitive dependency.
-If no project that contains the given UUID is found, the preference is recorded in the `Project.toml` file of the currently-active project.
-The initial state for preferences is an empty dictionary, package authors that wish to have a default value set for their preferences should use the `get(prefs, key, default)` pattern as shown in the code example above.
 
-## Compile-Time Preferences
+# A non-compiletime preference
+function set_username(username::String)
+    @set_preferences!("username" => username)
+end
+function get_username()
+    return @load_preference("username")
+end
 
-If a preference must be known at compile-time, (and hence changing it should invalidate your package's precompiled `.ji` file) access of it should be done through the `Preferences.CompileTime` module.
-The exact same API is exposed, but the preferences will be stored within a separate dictionary from normal `Preferences`, and any change made to these preferences will cause your package to be recompiled the next time it is loaded.
-Packages that wish to use purely compile-time preferences can simply `using Preferences.CompileTime`, mixed usage will require compile-time usage to access functions and macros via `CompileTime.@load_preferences()`, etc...
+end # module UsesPreferences
+```

src/Preferences.jl

@@ -1,21 +1,226 @@
 module Preferences
 
+using TOML
+using Base: UUID, TOMLCache
+
+export load_preference, @load_preference,
+       set_preferences!, @set_preferences!
+
+include("utils.jl")
+
+"""
+    load_preference(uuid_or_module, key, default = nothing)
+
+Load a particular preference from the `Preferences.toml` file, shallowly merging keys
+as it walks the hierarchy of load paths, loading preferences from all environments that
+list the given UUID as a direct dependency.
+
+Most users should use the `@load_preference` convenience macro which auto-determines the
+calling `Module`.
+"""
+function load_preference(uuid::UUID, key::String, default = nothing)
+    # Re-use definition in `base/loading.jl` so as to not repeat code.
+    d = Base.get_preferences(uuid)
+    if currently_compiling()
+        Base.record_compiletime_preference(uuid, key)
+    end
+    # Drop any nested `__clear__` keys:
+    function drop_clears(data::Dict)
+        delete!(data, "__clear__")
+        for (k, v) in data
+            if isa(v, Dict)
+                drop_clears(v)
+            end
+        end
+        return data
+    end
+    drop_clears(x) = x
+
+    return drop_clears(get(d, key, default))
+end
+function load_preference(m::Module, key::String, default = nothing)
+    return load_preference(get_uuid(m), key, default)
+end
+
 """
-    CompileTime
+    @load_preference(key)
 
-This module provides bindings for setting/getting preferences that can be used at compile
-time and will cause your `.ji` file to be invalidated when they are changed.
+Convenience macro to call `load_preference()` for the current package.
 """
-module CompileTime
-const PREFS_KEY = "compile-preferences"
-include("common.jl")
-end # module CompileTime
+macro load_preference(key, default = nothing)
+    return quote
+        load_preference($(esc(get_uuid(__module__))), $(esc(key)), $(esc(default)))
+    end
+end
 
-# Export `CompileTime` but don't `using` it
-export CompileTime
 
-# Second copy of code for non-compiletime preferences
-const PREFS_KEY = "preferences"
-include("common.jl")
+"""
+    process_sentinel_values!(prefs::Dict)
+
+Recursively search for preference values that end in `nothing` or `missing` leaves,
+which we handle specially, see the `set_preferences!()` docstring for more detail.
+"""
+function process_sentinel_values!(prefs::Dict)
+    # Need to widen `prefs` so that when we try to assign to `__clear__` below,
+    # we don't error due to a too-narrow type on `prefs`
+    prefs = Base._typeddict(prefs, Dict{String,Vector{String}}())
+
+    clear_keys = get(prefs, "__clear__", String[])
+    for k in collect(keys(prefs))
+        if prefs[k] === nothing
+            # If this should add `k` to the `__clear__` list, do so, then remove `k`
+            push!(clear_keys, k)
+            delete!(prefs, k)
+        elseif prefs[k] === missing
+            # If this should clear out the mapping for `k`, do so, and drop it from `clear_keys`
+            delete!(prefs, k)
+            filter!(x -> x != k, clear_keys)
+        elseif isa(prefs[k], Dict)
+            # Recurse for nested dictionaries
+            prefs[k] = process_sentinel_values!(prefs[k])
+        end
+    end
+    # Store the updated list of clear_keys
+    if !isempty(clear_keys)
+        prefs["__clear__"] = collect(Set(clear_keys))
+    else
+        delete!(prefs, "__clear__")
+    end
+    return prefs
+end
+
+# See the `set_preferences!()` docstring below for more details
+function set_preferences!(target_toml::String, pkg_name::String, pairs::Pair{String,<:Any}...; force::Bool = false)
+    # Load the old preferences in first, as we'll merge ours into whatever currently exists
+    d = Dict{String,Any}()
+    if isfile(target_toml)
+        d = Base.parsed_toml(target_toml)
+    end
+    prefs = d
+    if endswith(target_toml, "Project.toml")
+        if !haskey(prefs, "preferences")
+            prefs["preferences"] = Dict{String,Any}()
+        end
+        # If this is a `(Julia)Project.toml` file, we squirrel everything away under the
+        # "preferences" key, while for a `Preferences.toml` file it sits at top-level.
+        prefs = prefs["preferences"]
+    end
+    # Index into our package name
+    if !haskey(prefs, pkg_name)
+        prefs[pkg_name] = Dict{String,Any}()
+    end
+    # Set each preference, erroring unless `force` is set to `true`
+    for (k, v) in pairs
+        if !force && haskey(prefs[pkg_name], k) && (v === missing || prefs[pkg_name][k] != v)
+            throw(ArgumentError("Cannot set preference '$(k)' to '$(v)' for $(pkg_name) in $(target_toml): preference already set to '$(prefs[pkg_name][k])'!"))
+        end
+        prefs[pkg_name][k] = v
+
+        # Recursively scan for `nothing` and `missing` values that we need to handle specially
+        prefs[pkg_name] = process_sentinel_values!(prefs[pkg_name])
+    end
+    open(target_toml, "w") do io
+        TOML.print(io, d, sorted=true)
+    end
+    return nothing
+end
+
+"""
+    set_preferences!(uuid_or_module, prefs::Pair{String,Any}...; export_prefs=false, force=false)
+
+Sets a series of preferences for the given UUID/Module, identified by the pairs passed in
+as `prefs`.  Preferences are loaded from `Project.toml` and `LocalPreferences.toml` files
+on the load path, merging values together into a cohesive view, with preferences taking
+precedence in `LOAD_PATH` order, just as package resolution does.  Preferences stored in
+`Project.toml` files are considered "exported", as they are easily shared across package
+installs, whereas the `LocalPreferences.toml` file is meant to represent local
+preferences that are not typically shared.  `LocalPreferences.toml` settings override
+`Project.toml` settings where appropriate.
+
+After running `set_preferences!(uuid, "key" => value)`, a future invocation of
+`load_preference(uuid, "key")` will generally result in `value`, with the exception of
+the merging performed by `load_preference()` due to inheritance of preferences from
+elements higher up in the `load_path()`.  To control this inheritance, there are two
+special values that can be passed to `set_preferences!()`: `nothing` and `missing`.
+
+* Passing `missing` as the value causes all mappings of the associated key to be removed
+  from the current level of `LocalPreferences.toml` settings, allowing preferences set
+  higher in the chain of preferences to pass through.  Use this value when you want to
+  clear your settings but still inherit any higher settings for this key.
+
+* Passing `nothing` as the value causes all mappings of the associated key to be removed
+  from the current level of `LocalPreferences.toml` settings and blocks preferences set
+  higher in the chain of preferences from passing through.  Internally, this adds the
+  preference key to a `__clear__` list in the `LocalPreferences.toml` file, that will
+  prevent any preferences from leaking through from higher environments.
+
+Note that the behavior of `missing` and `nothing` is both similar (they both clear the
+current settings) and diametrically opposed (one allows inheritance of preferences, the
+other does not).  They can also be composed with a normal `set_preferences!()` call:
+
+```julia
+@set_preferences!("compiler_options" => nothing)
+@set_preferences!("compiler_options" => Dict("CXXFLAGS" => "-g", LDFLAGS => "-ljulia"))
+```
+
+The above snippet first clears the `"compiler_options"` key of any inheriting influence,
+then sets a preference option, which guarantees that future loading of that preference
+will be exactly what was saved here.  If we wanted to re-enable inheritance from higher
+up in the chain, we could do the same but passing `missing` first.
+
+The `export_prefs` option determines whether the preferences being set should be stored
+within `LocalPreferences.toml` or `Project.toml`. 
+"""
+function set_preferences!(u::UUID, prefs::Pair{String,<:Any}...; export_prefs=false, kwargs...)
+    # Find the first `Project.toml` that has this UUID as a direct dependency
+    project_toml, pkg_name = find_first_project_with_uuid(u)
+    if project_toml === nothing && pkg_name === nothing
+        # If we couldn't find one, we're going to use `active_project()`
+        project_toml = Base.active_project()
+
+        # And we're going to need to add this UUID as an "extras" dependency:
+        # We're going to assume you want to name this this dependency in the
+        # same way as it's been loaded:
+        pkg_uuid_matches = filter(d -> d.uuid == u, keys(Base.loaded_modules))
+        if isempty(pkg_uuid_matches)
+            error("Cannot set preferences of an unknown package that is not loaded!")
+        end
+        pkg_name = first(pkg_uuid_matches).name
+
+        # Read in the project, add the deps, write it back out!
+        project = Base.parsed_toml(project_toml)
+        if !haskey(project, "extras")
+            project["extras"] = Dict{String,Any}()
+        end
+        project["extras"][pkg_name] = string(u)
+        open(project_toml, "w") do io
+            TOML.print(io, project; sorted=true)
+        end
+    end
+
+    # Finally, save the preferences out to either `Project.toml` or `Preferences.toml`
+    # keyed under that `pkg_name`:
+    target_toml = project_toml
+    if !export_prefs
+        target_toml = joinpath(dirname(project_toml), "LocalPreferences.toml")
+    end 
+    return set_preferences!(target_toml, pkg_name, prefs...; kwargs...)
+end
+function set_preferences!(m::Module, prefs::Pair{String,<:Any}...; kwargs...)
+    return set_preferences!(get_uuid(m), prefs...; kwargs...)
+end
+
+"""
+    @set_preferences!(prefs...)
+
+Convenience macro to call `set_preferences!()` for the current package.  Defaults to
+setting `force=true`, since a package should have full control over itself, but not
+so for setting the preferences in other packages, pending private dependencies.
+"""
+macro set_preferences!(prefs...)
+    return quote
+        set_preferences!($(esc(get_uuid(__module__))), $(esc(prefs...)), force=true)
+    end
+end
 
 end # module Preferences