Julep: interlocked changes across the package ecosystem #770
Comments
|
This seems like it might be a case for a "development registry" where you can play around with versions until you get it right and everything works and then upstream the lot of them all at once. Would that work? One can also work on branches of a registry and tweak things until they're right. We can consider versions on branches not to be "written in stone" until they're merged into master. Relatedly, I've been thinking about tooling to allow orgs to have their own registries where they can implement their own CI policies and include whatever upstream dependencies they "allow", pulling those from the General registry. Then people submit versions to the org registry for tagging and the org registry can implement whatever CI policies that they care to. Testing, coding, compatibility standards, etc. If an org registry is trusted, then General can just pull versions from the org registry automatically. |
|
Yes, a development registry sounds like it would work well, and is better in the sense that it leverages existing infrastructure. The org-specific registry also sounds pretty cool. A fair amount of work to configure initially, perhaps, but for a big org with lots of activity I can easily see it being worthwhile. |
|
I've only recently transitioned to Julia 0.7 so I may be unaware of some of the available development tools, but I've been finding it cumbersome to do CI with a development registry. Say that package A depends on package B, and B depends on C. If I've made a set of local changes to all 3 packages and want to push the changes and let Travis run tests, it seems I need to take this sequence of actions:
whereas before Pkg3 the sequence was simply
Are there any tools to make this easier with Pkg3? Is my workflow in need of revision? Or is the thought that it's worth this extra development labor since it gives package users a more stable experience? In cases like this I'm already feeling tempted to just combine A, B, and C into one package (when they're my own packages) to avoid this extra work. Usually I think of that as worse development practice, but I wonder if others are feeling the same temptation. I'm hoping that someone will say "You're doing it wrong." rather than "Don't be lazy." :) |
|
You should be able to check in a Manifest.toml file that uses non-registered versions of dependencies and it should just work. There's no requirement that versions in manifests are registered anywhere. |
I just want to make sure I understand: Are you suggesting that I check a Manifest.toml into each of the repos for packages A, B, and C in my example? I think I understand how that would allow me to develop freely without worrying about updating versions, but then when it's time to make my updated packages available to others through a registry I think I would need more commits removing Manifest.toml from each package? Or are you suggesting to avoid using a registry altogether? In our case that doesn't seem ideal. |
|
There's no need to remove the |
Ah! I didn't understand this. Somewhere I read that |
|
Committing the Manifest isn't working for me. I suspect the problem is due to my script:
- if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
- julia -e 'using Pkg, LibGit2;
user_regs = joinpath(DEPOT_PATH[1],"registries");
mkpath(user_regs);
all_registries = Dict("General" => "https://github.com/JuliaRegistries/General.git",
"HolyLabRegistry" => "git@github.com:HolyLab/HolyLabRegistry.git");
Base.shred!(LibGit2.CachedCredentials()) do creds
for (reg, url) in all_registries
path = joinpath(user_regs, reg);
LibGit2.with(Pkg.GitTools.clone(url, path; header = "registry $reg from $(repr(url))", credentials = creds)) do repo end
end
end'
- julia -e 'using Pkg; Pkg.build(); Pkg.test("RegisterMismatch"; coverage=false)'In the Travis build log, there's a section Unfortunately it doesn't specify the |
|
I'll close this as a dup of #770 since they seem fairly similar. |
|
This is very similar to #770. |
|
😢 I always do this |
At quite a few points in my development experience I've made changes in one package that have ripple effects across many others. I remember one case involving colors and/or images where just the sequence of making a change in one package, waiting for tests to finish, merging and tagging, getting the tag merged to metadata (me "abusing" my METADATA merge privileges and doing it ASAP, without waiting for someone to review it), and then going on to the next package literally took an entire weekend doing little else, even though I already had local branches queued for all of the changes.
The latest (much smaller) version of this just happened with OrderedCollections/DataStructures, JuliaCollections/OrderedCollections.jl#10.
It would be lovely to have some way of easily expressing a transient dependency on not-yet-merged branches in other packages, without having any more customization than necessary that later needs to be backed out. My main suggestion for this would be to add a top-level file like
PreferredBranch.tomlwith contentsand list the specific branch that CI should check out in each package. This would have two impacts:
PreferredBranch.tomlfile to the next project, adding the relevant line about the right branch for the just-submitted PR in the previous package.PreferredBranch.tomlfile to attobot, which does everything (merging the PRs and submitting the METADATA tags) all at once; after merging to METADATA attobot also submits and merges a PR to deletePreferredBranch.tomlfrom each package.The text was updated successfully, but these errors were encountered: