Nimble migration tracker

[arnetheduck]

We're planning on migrating functionality from nimbus-build-system to nimble as the latter gains capabilities. This issue is an overview / tracker of open high-level issues on that road, and will be edited as we move along.

Blockers:

• Treat Nim itself as any other library - it should be subject to the lock file as well
• --noNimblePath should unconditionally be set for projects with lock files nim-lang/nimble#1004
• Choose version selection strategy
  • Minimal version (like go, minimises churn)
  • Semantic versioning (prevents upgrades from running past past known compatibility barriers)
• Resolve parent path issue - config.nims that contains the ugly hack for importing paths typically lives in a parent folder - this means we must never nest projects, or the search for "parent" paths needs to stop at the "first found" location

[zah]

Nesting projects is a bad idea in general due to config file pollution (the cited paths problem is just one example for this). We've had many occasions where the CI tests of a library were failing even when they pass locally for a developer due to unintended help from the top-level project's config file and also situations where we cannot run the tests of a library in the vendor folder due to unwanted interference from the top-level config file.

For these reasons, the nim-workspace repository is using a directory layout where all projects are placed next to each other (including the "top-level" projects). Another benefit of this layout is that it also makes it possible to share the same development check outs of multiple libraries while testing changes in them in multiple "top-level" projects.

[arnetheduck]

Well, one feature requirement we have is to support monorepo layouts - ie handling multiple git repositories will always be a major pita / UX problem no matter how much magic tooling you add on top - since the libraries under a monorepo are "related", there needs to be a structure in place for dealing with such cases, or rather, a way to declare "monorepo" repositories in a way that the package manager understands and sets up for a good out-of-the-box experience.

Most users won't be using nim-workspace (and shouldn't have to) - this is a feature that the tool needs to provide "out-of-the-box".

[arnetheduck]

ie I don't mind a "flat" layout solving this problem, but it is a problem that needs to be solved explicitly (for example by nimble giving an error on such layouts) because otherwise, users will end up with nested config files that interfere with each other.

This can happen by mistake or by simply trying to nest projects - it is specially bad with lock files in libraries because then the library code will be picking up both its own lock file config and that of the parent.

[zah]

Well, in what I described above, there are unintended interactions between the config files of the projects. And more importantly, there are differences in the configurations between the local builds and the CI builds.

In a monorepo, there will be only intended interactions between the config files and both the local builds and the CI builds will work with the same config files.

[arnetheduck]

In a monorepo, there will be only intended interactions between the config files and both the local builds and the CI builds will work with the same config files.

but you'll have the reverse problem for people that try to use one of the "sub-libraries" independently - ie the whole point of splitting up a monorepo into sub-libs is so that "outside" consumers can get away with a smaller dependency - thus, for monorepo to work, each library necessarily needs to be fully self-contained (or the library-in-monorepo extractor needs to understand that parent folders form a part of the library and replicate it accordingly)

[zah]

Yes, it would be up to the maintainers of the monorepo to ensure that the user experience of people cloning only subfolders is adequate. But still, any sharing they allow to happen would be intentional and will be potentially peer reviewed. And admittedly, Nim and Nimble can certainly provide helpful tools for maintaining more hygiene in such layouts.

It should be noted though that config files in libraries are used mostly for the development of the library. Users of the library in other projects generally don't consume the config files in any way.