supporting a build matrix

[michaelkirk]

Heads up that I am currently mucking around in the georust dockerhub in pursuit of having an array of containers to test support on different versions of rust.

I can think of several approaches, but as usual, I'm adrift in a the docker sea. It's not clear to me what the best way to organize a matrix of containers is, so I'm experimenting a bit. I'm intending not to break anything, but... let me know if I have.

This is kind of a free-form/discussion issue. I'm not blocked or anything, so don't feel like you need to read unless you're interested.

One piece of background context I'm learning is that conventionally docker "tags" aren't intended to be immutable like a "tag" in git, but instead refer to something more like a "release track" or a "semver" spec.

For example, the rust:1.45 tag can be read as "the latest rust container published for rust 1.45.*". Depending on when your container was built, this might have been 1.45.0, 1.45.1, or 1.45.2. (see https://hub.docker.com/_/rust?tab=tags&page=1&ordering=last_updated&name=1.45). This is by design apparently.

If you want to use a specific rust version only, you could pin to the 3 section tag for the patch version rust:1.45.1, or I suppose to the container sha if you're really paranoid.

So, running with that, it's entirely normal that I could publish geo-ci:rust-1.50, and then if rust has a 1.50.1 release, to rebuild and clobber the geo-ci:rust-1.50 tag with a new container which will be based on the rust:1.50.1 base container. We could track minor patch versions if we wanted, but that seems like overkill for now.

So, I'm currently setting up docker hub to build and tag e.g. geo-ci:rust-1.49, geo-ci:rust-1.50.

@urschrei - if you're ok with it, I'd like to also roll out this convention for the proj family of containers.

My understanding is that proj only officially supports one version (the latest) of libproj at a given time. So given that, I suppose we don't really need to include the proj version in the tag name, and can use the same proj-ci:rust-1.49, proj-ci:rust-1.50.

At such a time when proj officially supports a window of proj releases (which I think would be great!), the best scheme I can come up with would be a two dimensional tag like: proj-ci:proj-7.2.0-rust-1.49, proj-ci:proj-7.2.1-rust-1.49, proj-ci:proj-7.2.0-rust-1.50, proj-ci:proj-7.2.1-rust-1.50

If you'd prefer, I'm also OK with doing the two dimensional tagging now even with only one proj version in play, just to be really precise about what's being used where.

One day I'd also like to add a geo-ci:rust-nightly that rebuilds on a schedule. Probably we wouldn't want to fail CI's with it, but it seems like it'd at least be good to know.

---

I'm currently pursuing a file structure, I believe similar to what rust does for its own official containers, where each supported version has its Dockerfiles exist in a duplicated directory.

e.g.

.
├── README.md
├── rust-1.49
│   ├── geo-ci.Dockerfile
│   ├── libproj-builder.Dockerfile
│   ├── proj-ci-without-system-proj.Dockerfile
│   └── proj-ci.Dockerfile
└── rust-1.50
    ├── geo-ci.Dockerfile
    ├── libproj-builder.Dockerfile
    ├── proj-ci-without-system-proj.Dockerfile
    └── proj-ci.Dockerfile

An alternative I considered was to have a branch per-supported rust track. e.g.

branch: "supported-builds/rust-1.49"

├── README.md
├── geo-ci.Dockerfile
├── libproj-builder.Dockerfile
├── proj-ci-without-system-proj.Dockerfile
└── proj-ci.Dockerfile

branch: "supported-builds/rust-1.50"

├── README.md
├── geo-ci.Dockerfile
├── libproj-builder.Dockerfile
├── proj-ci-without-system-proj.Dockerfile
└── proj-ci.Dockerfile

Which I think would also work, but it felt less discoverable to encode essential operational information in branch structure.

[urschrei]

I'm about to go approve / thumbs up the PR, having read through the updated docs, which are great. I have one general, and one specific comment:

1. This feels like so much work! I think our current CI setup is excellent, and the addition of previous rustc is welcome, but it may be worth spending some time figuring out whether any more of the update steps can be automated (and what form that automation might take). Maybe I'm overthinking it, and 45 minutes of maintenance every six weeks is fine, and I'm happy for us to actually experience a couple of update cycles and then see how we feel before trying to automate anything
2. A window of proj versions might be difficult given that we bundle a version. We may be able to accept a window of installed libproj versions, but bundled-proj would always be latest stable. And we'd have to tweak our CI (more containers 🙃?) Still: interesting!

[michaelkirk]

This feels like so much work!

Historically I have spent an embarrassingly large amount of time on docker/ci related things, but 95% of my time is spent figuring out what to try and then watching CI TV as automated machinery runs. It's typically a slow feedback loop.

I'm hopeful that re-executing the steps of a known process will be a comparatively small thing. 🤞

I'm happy for us to actually experience a couple of update cycles and then see how we feel before trying to automate anything

That sounds good to me! I'm definitely open to automation if and when we have confidence the process itself makes sense, and willing to do carry the load of the manual process in the meanwhile. Feel free to tag me if there's a proj update you'd like to offload.