[feature] Lockfile 'base' with recipe revisions only workflow

[prince-chrismc]

• I've read the CONTRIBUTING guide.

Take the given conanfile.py snippet:

    def requirements(self):
        self.requires("restinio/0.6.10")
        self.requires("json-schema-validator/2.1.0")
        self.requires("lyra/1.4.0")
        self.requires("spdlog/1.8.0")

conan lock create

$ conan lock create conanfile.py --version=1.0.0-dev.0 --base --lockfile-out=conan-deps.lock
Requirements
    asio/1.17.0 from 'conan-center' - Cache
    expected-lite/0.4.0 from 'conan-center' - Cache
    fmt/7.0.3 from 'conan-center' - Cache
    http_parser/2.9.4 from 'conan-center' - Cache
    json-schema-validator/2.1.0 from 'conan-center' - Cache
    lyra/1.4.0 from 'conan-center' - Cache
    nlohmann_json/3.9.1 from 'conan-center' - Cache
    openssl/1.1.1g from 'conan-center' - Cache
    optional-lite/3.2.0 from 'conan-center' - Cache
    restinio/0.6.10 from 'conan-center' - Cache
    spdlog/1.8.0 from 'conan-center' - Cache
    string-view-lite/1.4.0 from 'conan-center' - Cache
    variant-lite/1.2.2 from 'conan-center' - Cache
Packages
    asio/1.17.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    expected-lite/0.4.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    fmt/7.0.3:66c5327ebdcecae0a01a863939964495fa019a06 - Download
    http_parser/2.9.4:d50a0d523d98c15bb147b18fa7d203887c38be8b - Download
    json-schema-validator/2.1.0:8de143647f7a68ba9847f010597556418f1f6953 - Download
    lyra/1.4.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    nlohmann_json/3.9.1:d1091b2ed420e6d287293709a907ae824d5de508 - Cache
    openssl/1.1.1g:d50a0d523d98c15bb147b18fa7d203887c38be8b - Download
    optional-lite/3.2.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    restinio/0.6.10:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    spdlog/1.8.0:72cd2c4cdabb80343b4cefc3b31911672f0e2551 - Download
    string-view-lite/1.4.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    variant-lite/1.2.2:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
Build requirements
    catch2/2.13.0 from 'conan-center' - Cache
Build requirements packages
    catch2/2.13.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache

Generated lockfile: backend/conan-deps.lock

Which results in the generate lockfile.

conan-deps.lock

{
 "graph_lock": {
  "nodes": {
   "0": {
    "ref": "user-management/1.0.0-dev.0",
    "requires": [
     "1",
     "10",
     "12",
     "13"
    ],
    "build_requires": [
     "14"
    ],
    "path": "conanfile.py",
    "context": "host"
   },
   "1": {
    "ref": "restinio/0.6.10#6ded525708af0f9298a32bffc35cf7f5",
    "requires": [
     "2",
     "3",
     "4",
     "5",
     "6",
     "7",
     "8",
     "9"
    ],
    "context": "host"
   },
   "2": {
    "ref": "http_parser/2.9.4#16c06558f4414108119e2ebf26e9aea6",
    "context": "host"
   },
   "3": {
    "ref": "fmt/7.0.3#2e4cfb84c7647e8c2b5fef1ebb527b7a",
    "context": "host"
   },
   "4": {
    "ref": "expected-lite/0.4.0#20b6eb044edb5f0554695de4f69bb4a2",
    "context": "host"
   },
   "5": {
    "ref": "optional-lite/3.2.0#16ff2c5bedc906252c92e04fb05b4d8c",
    "context": "host"
   },
   "6": {
    "ref": "string-view-lite/1.4.0#ba6c7fdd59c14dcd95fddc655f60e463",
    "context": "host"
   },
   "7": {
    "ref": "variant-lite/1.2.2#e2a9ee39933cb18d2326dab6cfe97375",
    "context": "host"
   },
   "8": {
    "ref": "asio/1.17.0#d5e050dc0e244955b037e5160071935a",
    "context": "host"
   },
   "9": {
    "ref": "openssl/1.1.1g#7db1aa62818c6bb1a1e51166f8fb602d",
    "context": "host"
   },
   "10": {
    "ref": "json-schema-validator/2.1.0#a2a0f16dc456d099ebea16329868c60d",
    "requires": [
     "11"
    ],
    "context": "host"
   },
   "11": {
    "ref": "nlohmann_json/3.9.1#ea6e1cd91366a6aeebad1f31213bccd2",
    "context": "host"
   },
   "12": {
    "ref": "lyra/1.4.0#7f8cb9aa1b30b40773537d1a991dda52",
    "context": "host"
   },
   "13": {
    "ref": "spdlog/1.8.0#a4a3e8f6f3cd5b62992585ca9eb50dfc",
    "requires": [
     "3"
    ],
    "context": "host"
   },
   "14": {
    "ref": "catch2/2.13.0#b8f4e03bfc860a975393ffd815ba7a46",
    "context": "host"
   }
  },
  "revisions_enabled": true
 },
 "version": "0.4"
}

Just image if

You could run conan install .. -s build_type=Debug --lockfile=../conan-deps.lock and you can get a reproducible dependency graph every time that is tracked by git.

However currently,

$ conan install .. -s build_type=Debug --lockfile=../conan-deps.lock
ERROR: Cannot use profile, settings, options or env 'host' when using lockfile
$ conan install .. --lockfile=../conan-deps.lock
ERROR: Lockfiles with --base do not contain profile information, cannot be used. Create a full lockfile

This is an experimental feature but I was hoping for an explanation about the current limitation and if there's any road map for expanding this feature to help address the short coming regarding upstream changes breaking consumer. It looks to have a lot of potential

Thank you in advance for taking time to answer.

[memsharded]

Hi @prince-chrismc

If I understood the issue correctly, then this flow is possible, but just not in one step, but in 2 steps. You need first to compute the complete lockfile, then do the conan install:

$ conan lock create .. --lockfile=../conan-deps.lock --lockfile-out=../conan-deps-debug.lock -s build_type=Debug
$ conan install .. --lockfile-out=../conan-deps.lock

While doing it in 1 step would be theoretically possible, it would be very difficult in practice to clearly differentiate what is a change due to the fact that the input lockfile is "base" only, and what comes from a result of building from sources. It seems conceptually easier to manage the following rules:

• A lockfile can never modify its topology, add or remove packages or change any of its value when it is being used (conan install, create). If something tries to change it, it will error.
• Lockfile evolution is done with conan lock create command, deriving a new lockfile from an existing one.

Conan 1.30 will add a bit more of docs about this point, but please let me know if this clarifies it a bit.

[prince-chrismc]

this flow is possible, but just not in one step, but in 2 steps. You need first to compute the complete lockfile, then do the conan install

The complete lock file is the "issue". I would love to have the partial or base lock file since it does not constrained to the host, compiler, options or build settings.

I currently need to RREV for all deps within my tree to ensure we are not impacted by changes upstream listed in my conanfile.txt.

Having a minimum conanfile.txt with only top level dependencies, and no recipe revision would help take off an overhead on developers.

This would enable reproducible installation of exact topology from a fixed set of top level dependencies regardless of the toolset.

I think a good analog would be package.json and yarn.lock with nodejs development.

• A lockfile can never modify its topology, add or remove packages or change any of its value when it is being used (conan install, create). If something tries to change it, it will error.

That's perfect. It will prevent upstream changes impacting CI pipelines.

• Lockfile evolution is done with conan lock create command, deriving a new lockfile from an existing one.

That's simple enough. It's easy to specify an input and output file as an upgrade path.

[prince-chrismc]

I feel I am not be expressing the two different types of lockfiles.

[memsharded]

The complete lock file is the "issue". I would love to have the partial or base lock file since it does not constrained to the host, compiler, options or build settings.

I currently need to RREV for all deps within my tree to ensure we are not impacted by changes upstream listed in my conanfile.txt.

This is impossible in the general case. As the dependency graph can be conditional (and in practice, this happens relatively often), it is not possible that a single "base" lockfile can ensure locking of absolutely everything in all platforms.

The "base" lockfile aims to this, and if there is no variability in the graph, it will achieve this behavior.

I think a good analog would be package.json and yarn.lock with nodejs development.

It is probably easier to lock in the js ecosystem, because the variability of the dependency graph is not that common, and they can assume other working hypothesis (as having multiple versions of the same package as dependencies in the same dependency graph)

I feel I am not be expressing the two different types of lockfiles.

Could you please clarify what you mean? Isn't the "base" + "full" lockfiles a good enough solution? You can treat the full lockfiles as temporaries, and not store or manage them, just compute and use them, and manage the "base" lockfile only. Please let me know, many thanks for the feedback!

[prince-chrismc]

Thanks for adding more explanation, I think I am starting to understand your original reply.

Let's see if I can explain this back.

1. Have a conanfile.txt with only the direct dependencies up to version specific
   • this would still require me to resolve conflicts (reasonable)
   • this still gives the flexibility to override versions manually (ie openssl)
2. Create a "base" lock file from conan lock create --base
   • this will contain the full RREV topography
   • platform and toolset agnostic
   • could be checked into source control
3. On demand create a full lock via conan lock create <input> --out <output>
   a. For developers this is an easy step that can be saved and copy pasted
   b. Very easy to add one line to CI build script
4. Run the normal conan install referencing the "full" lockfile

Workflow:

$ conan lock create .. --lockfile=../conan-deps.lock --lockfile-out=../conan-deps-debug.lock -s build_type=Debug
$ conan install .. --lockfile=../conan-deps-debug.lock

[prince-chrismc]

Firstly I'd like to take the time to thank you for explaining this topic to me. ❤️ It's greatly appreciated!

I have been successful in implementing what you have described, you can see the PR linked is now merged!

I am going to watch how it reacts for a little bit before possibly changing the "best practices" at work since this helps cut some of the over head I'm getting complaints about.

I would love to close this issue, but I think reviewing the documentation and contributing back any suggestions is a requirement on my part.

Quickly searching the repo there's only 17 hits. I know there's a "How to" section and I would love to add one for this, it's not obvious (for those intermediate Conan users like myself) this workflow exists... Would that be welcomed from the Conan team?

[memsharded]

Hi @prince-chrismc

Thanks for telling, I am very happy that you were successful implementing it!

Would that be welcomed from the Conan team?

Of course! Please let us know with some more detail how do you think this could be explained better in the docs, or contribute something yourself, as you wish.

I am fine with keeping it open, if at this point is more a documentation issue, let's move this issue there, and you can open new issues for the tool if code changes are necessary. Thanks for the feedback!