My plan was to add another method to the RPC that given the project model returns all the source dependencies that should be build.

We can then create a graph of which source dependencies depend on other source dependencies and create a topological ordering of which package should be processed before the others.

Then when calling getMetadata the repodata records for the source records are also passed along (requires that getMetadata is also called for the source dependencies it depends on).

When calling build this should also happen (and the packages need to be available from a local channel).

Alternatively: We could also consider a different approach where a build backend explicitly asks pixi (through RPC) to solve and environment including source dependencies. Pixi can then figure out what to build when but more importantly cache a lot of information about source dependencies.

Build backends could also ask pixi to install an environment including source dependencies..

@baszalmstra, @tdejager and I discussed the two approaches and decided for the one where:

a build backend explicitly asks pixi (through RPC) to solve and environment including source dependencies.

The expectation is that this makes things easier in the long run, and mid-term helps with caching information about source dependencies. It would also allow us to store build and host environments of source dependencies in the lock file.

The flow diagram would look like as follows.
Note that build-backends call a remote-procedure on pixi. We will have to set up infrastructure for that.
This procedure solve will have to be called for solving both build and host environment.
Pixi will then return the solved environment by passing repodata records which will then be passed to rattler-build via the finalized_dependencies parameter: https://github.com/prefix-dev/pixi-build-backends/blob/bb29d9f5a2b32a61608cb016c17a6c9ed149862b/crates/pixi-build/src/bin/pixi-build-cmake/protocol.rs#L118

This is a pretty big refactor, but luckily it can be done in multiple steps. As a first step, we will implement the RPC infrastructure to let the backend call pixi, implement solve and then pass the solved environment to rattler-build.
However, in that first step, we will not deal with recursive source dependencies at all.

After another discussion we decided on the following approach after all:

My plan was to add another method to the RPC that given the project model returns all the source dependencies that should be build.
We can then create a graph of which source dependencies depend on other source dependencies and create a topological ordering of which package should be processed before the others.
Then when calling getMetadata the repodata records for the source records are also passed along (requires that getMetadata is also called for the source dependencies it depends on).
When calling build this should also happen (and the packages need to be available from a local channel).

• Call getMetaDataSourceDeps recursively in order to find out which source dependencies need to be built and in which order (for the order only build/host dependencies are relevant, runtime are the only ones that will end up in the environment)
• Calling getMetadata/build in the correct order

Work:

• Add getMetaDataSourceDeps procedure
• Extend getMetadataParams & BuildParams in order to be able to pass repodata_records (of the source dependencies the packages depends on)
• Find out how to convince rattler build to take these repodata_records into account

The flow chart also applies to build, with getMetadata replaced with build

In a follow-up meeting, we clarified more aspects about the implementation:

• metadata call should handle run exports and dependencies
• build and host environments should to be explicitly defined and locked for reproducibility

We came up with a sample how to lock file might look like:

source:
- location: .
  build-backend:
    linux-64:
      - conda: https://prefix.dev/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
      - conda: https://prefix.dev/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2
      - conda: https://prefix.dev/conda-forge/linux-64/attr-2.5.1-h166bdaf_1.tar.bz2
      ...
    win-64:
      ...
    # All supported platforms will be here.

  outputs:
    - name: sdl_example
      version: 0.1.0
      build: hbf21a9e_0
      subdir: linux-64
      depends:
      - libstdcxx >=14
      - libgcc >=14
      - sdl2 >=2.30.10,<3.0a0
      input:
        hash: 2e5604492829c35ebd9881969780b23c269d5a4133bfe44993cf6545cf0c8b0b
        globs:
        - pixi.toml
      environments:
        host:
          linux-64:
          - conda: https://prefix.dev/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
          - conda: https://prefix.dev/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2
          - conda: https://prefix.dev/conda-forge/linux-64/attr-2.5.1-h166bdaf_1.tar.bz2
          ...
          # win-64: could also be there, but since we never cross-compile we dont have it. But for a noarch package we might.
        build:     
          linux-64:  
          - conda: https://prefix.dev/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
          - conda: https://prefix.dev/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2
          - conda: https://prefix.dev/conda-forge/linux-64/attr-2.5.1-h166bdaf_1.tar.bz2
          ...
          # win-64: could also be there, but since we never cross-compile we dont have it. But for a noarch package we might.

        # Environment names are not schemad?
    - name: sdl_example
      version: 0.1.0
      build: hbf21a9e_0
      subdir: osx-64
      depends:
      - libcxx >=19
      - sdl2 >=2.30.10,<3.0a0
      input:
        hash: 2e5604492829c35ebd9881969780b23c269d5a4133bfe44993cf6545cf0c8b0b
        globs:
        - pixi.toml
    - name: sdl_example
      version: 0.1.0
      build: hbf21a9e_0
      subdir: osx-arm64
      depends:
      - libcxx >=19
      - sdl2 >=2.30.10,<3.0a0
      input:
        hash: 2e5604492829c35ebd9881969780b23c269d5a4133bfe44993cf6545cf0c8b0b
        globs:
        - pixi.toml
    - name: sdl_example
      version: 0.1.0
      build: hbf21a9e_0
      subdir: win-64
      depends:
      - vc >=14.1,<15
      - vc14_runtime >=14.16.27033
      - sdl2 >=2.30.10,<3.0a0
      input:
        hash: 2e5604492829c35ebd9881969780b23c269d5a4133bfe44993cf6545cf0c8b0b
        globs:
        - pixi.toml