Automated version bounds

[bramtayl]

I've been having a lot of trouble recently finding version bounds for my packages. It seems like this could be automated. A brute force way would be to run Pkg.test for the most recent version of each dependency, and then slowly move back the versions until Pkg.test fails. Of course, this would take forever for big packages with big dependencies, but it would be worth it for safety. Maybe this belongs in a separate package, but something like this seems key to the stability of the package ecosystem.

[tkelman]

Making a tool that does the simple thing (with some selectable layers of isolation for testing different versions, and outputting a summary markdown table) has been on my to-do list for a while. Open to collaboration and design discussion to build something. Don't think it needs to be part of PkgDev though.

[bramtayl]

The only real issue I can think of is the order in which you step the packages back, which seems like it could lead to different outcomes in some cases.

[tkelman]

cc also @ararslan who may have had some ideas at https://github.com/ararslan/Bounder.jl

[ararslan]

Unfortunately Bounder neither works (last I checked, which was a long time ago) nor does it do anything particularly interesting; it only sets version bounds in past tags to a manually specified version.

The only way I could think to automate the setting of version bounds would be to have something like JuliaCIBot, which runs on METADATA, actually perform the combinatorial explosion of Pkg.pins and Pkg.tests and report the oldest set of dependency versions for which the package's tests pass.

[ararslan]

Though that quickly becomes infeasible for big packages such as Distributions, whose test suite takes upwards of 20 minutes to run and has a fair number of dependencies.

[bramtayl]

I'm not sure every combinatorial possibility needs to be tested, provided we assume some weak Pareto constraint. That is, if a certain set of dependency versions works, any set with all dependencies equal or higher will also work. Then we could just lower each dependency individually until it breaks. We could provide a fast and a slow version that relies on this constraint. Most of my packages would take very little time to test if this goes through.

[tkelman]

Yeah one-variable-at-a-time should be pretty good for getting things close. I don't think bounds have to be absolutely perfect, but it's worth using them to rule out combinations that obviously won't work.

Bounder is mostly just a good name for this, I think you were aiming for a different task with the code that's there (closer to what #94 does?).

[ararslan]

Yep, Bounder's intent was pretty much exactly #94. I'd be fine ceding the name to a better project, as my incarnation of Bounder is pretty much useless.

[bramtayl]

extract_requirements(args...) = begin
    path = joinpath(args...)
    if ispath(path)
        path |> Pkg.Reqs.parse |> keys
    else
        []
    end
end

"Make sure to remove all package bounds from base and test requires files first.
Makes the assumption that if a certain profile of versions work, all profiles
with versions greater or equal will also work."
minimum_requirement_versions(package_name, package_directory = Pkg.dir()) = begin
    package_file = joinpath(package_directory, package_name)
    requirements = setdiff(union(
        extract_requirements(package_file, "REQUIRE"),
        extract_requirements(package_file, "test", "REQUIRE")
    ), ["julia"])

    version_numbers = map(requirements) do requirement
        versions = VersionNumber.(
            joinpath(package_directory, requirement) |> 
            LibGit2.GitRepo |> 
            LibGit2.tag_list)
            
        while length(versions) > 1
            try
                Pkg.pin(requirement, versions[end - 1])
                Pkg.test(package_name)
                pop!(versions)
            catch
                break
            end
        end
        last_version = last(versions)
        Pkg.pin(requirement, last_version)
        last_version
    end
    Pkg.free.(requirements)
    Dict(zip(requirements, version_numbers))
end

[bramtayl]

Worth a new package?

[bramtayl]

I've got RequirementVersions.jl up and running. Tests pass locally but fail on travis. I thought might have been an issue with resolve but I added a custom version of pin that doesn't resolve and it didn't help. https://travis-ci.org/bramtayl/RequirementVersions.jl/jobs/264823671 @tkelman any ideas?

[tkelman]

Forgot this, sorry. Can we reconstruct exactly what sequence of commands is happening there? It might make more sense if we can isolate it to what Pkg is trying to do and see why it's going wrong.

[bramtayl]

Nevermind I think I need to disable the resolving that happens with Pkg.test as well