Add Multi repo support

[dosper7]

In several companies that use microservices have their own cloud platform teams and normally this "translates" into having a service per git repos.

In this cases how to use .NET Aspire? Meaning, different projects from different repos? It's not common to have multiple services within the same repo (unless you have a monorepo like Google 😆)

What is your suggestion?

[davidfowl]

Clone multiple repos locally, create an Uber solution and add the apphost to it, wiring up all of the projects required.

That's where I would start.

it you don’t want to pull and run the other repos locally, then I’m not sure exactly what the workflow would be (point to something already running).

[DamianEdwards]

Some patterns I imagine might emerge:

• git sub-modules for service repos and a parent repo with the .sln file and AppHost project, if you want to build and run all services locally
• Services published as container images that are then composed into an AppHost project as containers, if you don't want to build the services locally but you do want to run them locally
• Dev-instances of services deployed to dedicated environments and/or locally that are then composed into an AppHost project as HTTP endpoints, if you don't want to build or run the services locally

[bbrandt]

I believe we would need to publish our service defaults as a NuGet package, published to our own private NuGet server (Azure Artifacts in our case). Then projects throughout all the different repos could add a reference to this service defaults NuGet package.

[DamianEdwards]

I believe we would need to publish our service defaults as a NuGet package, published to our own private NuGet server (Azure Artifacts in our case). Then projects throughout all the different repos could add a reference to this service defaults NuGet package.

Yep, or you could put the ServiceDefaults in a sub-module and include it that way. There are lots of ways for folks to share code and assets among teams and I don't think we want to be too prescriptive about it. We should ensure we're flexible enough to support different patterns.

[artyom-p]

@DamianEdwards would be nice to have some samples on how to proceed with multi-repo scenarios. It is definitely a very cool and useful tool, but a huge part of microservice solutions are split across different repos, and in this case its not that obvious how to utilize it in local environment.

[dosper7]

I also don't understand how the deployment story is in this scenario, where we "run" our k8s cluster with a custom cloud platform team.

For instance, we use github + tekton pipelines + argo CD to deploy to k8s using multiple repos (one per service), all things k8s related are generated/managed by the platform (I've worked in previous companies with similar setup bu using different tools like gitlab as an example)

How Aspire can be helpful in this cases?

[davidfowl]

@artyom-p

@DamianEdwards would be nice to have some samples on how to proceed with multi-repo scenarios. It is definitely a very cool and useful tool, but a huge part of microservice solutions are split across different repos, and in this case its not that obvious how to utilize it in local environment.

Are you cloning everyone's repositories and running them today?

[davidfowl]

@dosper7

I also don't understand how the deployment story is in this scenario, where we "run" our k8s cluster with a custom cloud platform team.

It sounds like you don't need to use the deployment part of aspire because you have already figured it out! It's still great for local development, using components, and service discovery!

[artyom-p]

@davidfowl Yes, we clone all the repositories locally and debug service communication having multiple studio instances opened. Its doable, but its a pain.

[davidfowl]

@artyom-p What does the pain free solution look like? What would you ideal experience be?

[artyom-p]

@davidfowl Let's say we have 2 microservices that need to communicate with each other, they live in their own git repositories.

Each microservice can have its own or shared dependencies, for example:
ServiceA depends on ServiceB, Redis and MongoDB
ServiceB depends on MongoDB

There are cases when you want to debug only one service, but you still need another one to be running with all its dependencies, in this case, we can just somehow reference that second service and make it run, I think there was something similar in Tye before if I'm not mistaken.

The second case is more complex - when you want to debug both services. I'm not sure if it's doable without referencing projects of ServiceB in ServiceA solution.

[DGAISmith]

We also have the same multi-repo approach here, and this is where Tye was really strong for local development. A tye.yaml file could be relatively pathed to target x services, all cloned locally. Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire looks so much better in many ways to Tye, and I can see it's trying to achieve different things, but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

[davidfowl]

There are cases when you want to debug only one service, but you still need another one to be running with all its dependencies, in this case, we can just somehow reference that second service and make it run, I think there was something similar in Tye before if I'm not mistaken.

You can do a similar thing with the app host. Clone multiple repositories and create a project file that references individual projects. Maybe we need to enable referencing app hosts themselves 🤔.

Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire does require .NET 8 and above.

but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

Can you show me your tye setup?

[DGAISmith]

Many thanks for the reply, really enjoying the Conf this year.

Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire does require .NET 8 and above.

You say that, and of course you're totally correct if you want to use everything in the Aspire stack (and who am I to argue with the mad professors who came up with this amazing stuff), but I have been able to hack something together today from the starter application template using AddExecutable():

var builder = DistributedApplication.CreateBuilder(args);

//var apiservice = builder.AddProject<Projects.AspireApp2_ApiService>("apiservice");
var apiservicePort = 5323;
var apiserviceScheme = "http";
var apiserviceHostname = "localhost";

var apiservice = builder
    .AddExecutable("apiservice", "dotnet", @"..\AspireApp2.ApiService6", "run", $"--urls={apiserviceScheme}://{apiserviceHostname}:{apiservicePort}")
    .WithServiceBinding(apiservicePort, apiserviceScheme)
    .WithOtlpExporter();

builder.AddProject<Projects.AspireApp2_Web>("webfrontend");
    // .WithReference(apiservice);

builder.Build().Run();

In this case, ApiService6 is a dotnet6 api, inside which I've added OpenTelemetry support based on the code in the ServiceDefaults project. You do then end up with traces and (partial) metrics in the Asipre dashboard. So this is potentially useful-ish (I should try this with func.exe as well to see if we can spin up Functions in a decent way while Function support isn't quite ready yet). Service discovery is obvs broken here, and Aspire Components are presumably off the table too.

This serves two purposes though - allows some use of older dotnet applications, and also allows targeting of different repos outside of the solution the Aspire AppHost lives in. So it falls back to a similar kind of experience to Tye in these respects.
Probably not what you had in mind when you designed Aspire of course, but it's really nice to get some of the new dashboard experience with older projects.

but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

Can you show me your tye setup?

Sure, we have a lot of tye files, each targeting a specific surface area of our microservices (which are all in individual repos). Redacted example below:

name: Portal Backend Frontend
extensions:
- name: zipkin
- name: seq
services:
- name: Portal-UI
  executable: cmd
  args: '/c npm start'
  workingDirectory: ./MyCompany.Portal/src/Portal
  bindings:
    - host: localhost
      port: 80
      protocol: https
- name: Identity-Api
  project: ./MyCompany.IdentityApi/src/MyCompany.IdentityApi/MyCompany.IdentityApi.csproj
  bindings:  
   - port: 5000
     protocol: https	  
- name: Portal-Api
  project: ./MyCompany.PortalApi/src/MyCompany.PortalApi/MyCompany.PortalApi.csproj
  bindings:  
   - port: 19081
     protocol: http
- name: DataPull-Function
  azureFunction: ./MyCompany.Functions/src/MyCompany.DataPull.Functions
  bindings:
  - port: 7071
    protocol: http      
- name: DataMapper-Function
  azureFunction: ./MyCompany.Functions/src/MyCompany.DataMapper.Functions
  bindings:
  - port: 7072
    protocol: http      

Key thing here is that we haven't made any code changes to any of these services, and they're spread over 4 different repos.

[DamianEdwards]

@artyom-p I've opened dotnet/aspire-samples#35 to track adding some samples of multi-repo orchestration from an AppHost project.

[ZeBobo5]

If you have multiple repos, why don't you use GIT submodules in your App Host?

[DGAISmith]

If you have multiple repos, why don't you use GIT submodules in your App Host?

A reasonable question - I think this depends a lot on how isolated and decoupled you want to keep your microservices. There's also a point about simplicity - a tye file can reference an entire service with one entry (a few lines of yaml), but a submodule inside a super solution with all projects needing to be referenced is more complex and fragile WRT internal changes inside each targeted service.

Not to say this isn't an option, and it might be the best one on the table at the moment, but if Aspire could also support an "external orchestration" model then there's a more natural migration path from tye, and would be more flexible (but less feature-rich) by targeting older versions of dotnet. It might not be valuable enough for the work to be done though.

[davidfowl]

There's no magic here, it really comes down to how you want to run and reference your dependencies:

1. As container images
   • These could be images deployed to a registry
   • These could be tar files shared somewhere that can be loaded into your local docker
2. As a reference to remote urls (shared development environments? Sounds sketchy)
3. As source code (either docker files or projects)

For number 3, you need a local copy of the sources. That means something needs to git clone. We have to explore these approaches and see what the rough edges are to figure out what we can make easier about this experience.

I'm very interested in learning what people are doing today when you need to run locally, but have a dependency on another team's service(s). What do you do right now? What is painful about it? What could we do to improve on that pain?

[alefranz]

I'm adding my thoughts here, as this seems to be the megathread on the topic. Certainly, debugging remote URLs is challenging, similar to the issues faced with published images using current tooling. Moreover, running everything locally is not scalable and can quickly become slow, given the limitations of PC resources.

Probably, the sweet spot is to have closely related services, such as APIs within the same domain in a DDD context, run locally, and for other dependencies to point to "dev" instances deployed elsewhere.

I think a feature that allows explicit registration of external dependencies in the AppHost could be useful. To start, it would allow it to show up in the Dashboard, and then I'm sure there are other behaviors it can drive.

The main problem is that it will break the observability story, and adapting the dashboard would probably go beyond what you envision as scope.

[DamianEdwards]

We're looking to add support for referencing already hosted HTTP services in the AppHost project in #775

The point about this hosting model breaking the observability story (because the remote services' traces don't go into the local dashboard's OTLP endpoint) is a good one, but feels like a trade-off you need to make if you're not going to host the services locally.

[mumby0168]

I am very keen on the progression of this issue, and have a few bits that I have been thinking about.

I work in a very large micro services solution, we often try and keep communications between services via a message broker, however, we do have cases where we have a service that may take a HTTP dependancy for example.

I'd see our solution as being to large to run as one, but we'd maybe want to run parts of it with some set dependancies.

I wonder if that offers the opportunity of having multiple app hosts for a different set of services set up with there dependancy for the great local experience that aspire offers.

I think this may have already been mentioned in one of the sessions, but it could also be that a service is added to the app host just as a raw docker image hosted in a private registry, if it's the case you didn't want to debug the source for that solution, maybe?

The idea of packaging up the service default is also something I've been considering, is that something that the aspire team seem a reasonable idea?

[oleggolovkov]

For local development I was able to use it for multi-repo scenario with simple reverse engineering of AddProject: you should just implement an interface like this for each of your services:

public class MyService1 : IServiceMetadata
{
    public string AssemblyName => """MyService1""";
    public string AssemblyPath => """C:\src\root\MyService1\obj\Debug\net7\ref\MyService1.dll""";
    public string ProjectPath => """C:\src\root\MyService1\MyService1.csproj""";
}

and then you can do builder.AddProject<MyService1>("my service 1") so now instead of a bunch of open IDEs or a BAT file doing multiple dotnet run I only have this single project with apphost which starts A LOT faster than when running all of them separately and also has all those nice UIs with traces. Note that this requires no extra tools installed or changes to original projects

[dosper7]

I've just learned today about radapp.io, is there any kind of association between .NET aspire and radapp.io? It seems a potential great marriage story for dev local development -> to "any environment".

[davidfowl]

#811 (reply in thread)

[luizbon]

I'm a bit late to the party here.
This was something that came to my mind from the first time I saw Aspire.
Having the AppHost and all other services as code is great, but putting dependencies between them doesn't seem scalable, especially if you have a complex microservice solution. And this is where Open Telemetry is important.

If the tooling could provide a remote registration it would be nice.
From what I could see, there's a manifest generated and parsed.
The AppHost could expose an endpoint where services could self register by asking the host to download the manifest from their endpoints.

Disregarding security concerns, this approach could provide a disconnected way to register services.
And can even be expanded to other languages, in a similar way of the NodeJs example.

[davidfowl]

If the tooling could provide a remote registration it would be nice.

Can you walk me through the end to end here? Let's take a scenario with 2 teams, team A and B working on different services. B needs to call A at runtime (I assume this is the scenario we care about right?).

[luizbon]

Can you walk me through the end to end here? Let's take a scenario with 2 teams, team A and B working on different services. B needs to call A at runtime (I assume this is the scenario we care about right?).

Sure, let me try to explain.

• Team A adds the registration code to their service A, making the service self-register in the App Host.
  • Service A registration includes a URL to receive requests.
• Team B adds the registration code to their service B, making the service self-register in the App Host.
  • Service B registration includes a dependency on service A.

There are no edge cases if Service A is registered before Service B.
If Service B is registered before Service A, the host will tell Service B that Service A does not exist and that Service B should know how to handle that missing part. (ie. hold messages that depend on service A, or disable features based on this dependency) - The stack can provide a flag letting the app know about this lack of connectivity.

If we only think of service discovery solutions, this is nothing more than what Dapr already do, so there's no need to reinvent here besides moving away from the YAML file configurations, enabling self-registration and removing the sidecar.
The App Host can be a self-contained solution on top of Dapr, adding the extra functionalities from Aspire.

[davidfowl]

Still missing lots of details. Is the apphost running somewhere that app this team A and B are able to "register" with the app host? Is each developer supposed to do this or does this happen once. Is the app deployed or is it running on a developer's machine?

Is team B running team A's code? How does that work?

[luizbon]

Still missing lots of details. Is the apphost running somewhere that app this team A and B are able to "register" with the app host? Is each developer supposed to do this or does this happen once. Is the app deployed or is it running on a developer's machine?

Is team B running team A's code? How does that work?

My idea was more of a concept, not an actual definition.
But based on your questions, here is what I think.

AppHost is a service that can run locally or in a centralised environment.
If running in a central place, each team can connect to the same host and execute the registration of each service. This means that each team would have their services up and running for each other. This could be an integration testing environment.
Developers can hook their development versions to that host and work like dev containers. Since Aspire controls service discovery, it can also control routing to multiple instances and versions like a service mesh solution.

Another option would be to have the AppHost running locally. In this case, developers will need to register each other teams service.
For this, it could have two options: team A grabs the manifest from team B and vice-versa, just like an OpenAPI document. Then, register in their local AppHost and create a mock server. (Another feature Aspire can provide)
The second option would be to download a version of the other team's app via a package/docker image or the code itself. And proceed with regular service registration.

Each approach will depend on how development teams like to work, but the important thing is the stack will not impose how teams should work.
Aspire wants to be opinionated, but having the flexibility enables current solutions to integrate with Aspire and use the opinionated version for green field solutions.

[davidfowl]

I've added #1071 so far to make it a bit easier to this (#829 (comment)). Paths are relative to the app host project to make it easy to specify relative paths to the project. This should enable more experimentation.

[davidfowl]

My idea was more of a concept, not an actual definition.

I hear you, but I think we need to explore something in the realm of the feature set of the app host. The app host is never deployed, it's a local orchestrator only. It can run executables, containers, projects, cloud and reference remote resources (deployed elsewhere). The dashboard is also an OTLP server so telemetry is pushed to it with a single env variable config. With those primitives, it should be possible to build various workflows with these primitives.

We're still working through a couple of issues like enabling docker networking and making it easier to reference things within containers (so you can run your another team's container images for e.g.). Hopefully that'll be enough for you all to experiment and come back with feedback about what works and what doesn't work.

AppHost is a service that can run locally or in a centralised environment.
If running in a central place, each team can connect to the same host and execute the registration of each service. This means that each team would have their services up and running for each other. This could be an integration testing environment.
Developers can hook their development versions to that host and work like dev containers. Since Aspire controls service discovery, it can also control routing to multiple instances and versions like a service mesh solution.

I think we're pretty far away from this based on the current set of capabilities. That sad, maybe you could build something on top of these primitives to get the next level of understanding for what this workflow looks like.

[luizbon]

The app host is never deployed, it's a local orchestrator only.

That makes complete sense.
I want aware of this premise. Not sure if it is documented and I missed that.

Regarding my idea, I don't think it is aligned with the project goals then, but also can be implemented as an extension on top of it.

[davidfowl]

Regarding my idea, I don't think it is aligned with the project goals then, but also can be implemented as an extension on top of it.

Maybe consider other interesting ways to connect services like using an external service discovery system (dapr, consul) and the app host to bridge local and remote development.

Maybe you can use a shared configuration server (something like azure app config in the azure case) as a way to store service discovery information across teams.

The problem is the code needs to be executed (that means access to another teams' code or assets), or it needs to be running remotely.

[atrauzzi]

👋 -- Hello! Chiming in late here, but I was thinking: Why not make it possible to make a reference a git repo URL? With something like that, Aspire could clone and then either discover or be explicitly told what type of project is within for it to manage?

This would be a game changer, particularly for onboarding scenarios. I've followed a pattern of having a "home" repo that houses a docker-compose.yml definition that presumes the presence of other repositories in the past. Aspire is in a great position to improve on such a concept.

[dosper7]

@atrauzzi that seems good idea (just don't know about the execution part), but would definitely be helpful since the repos url is "always" there. It seems an approach like Go or Deno do with the "usings/imports".

Maybe something like this pseudo code:

var builder = DistributedApplication.CreateBuilder(args);
var apiservice = builder.AddGitProject("www.github.com/FooCompany/BaaRepoName").WithName("apiservice");

What do you think? Crazy stupid idea?

[davidfowl]

👋 -- Hello! Chiming in late here, but I was thinking: Why not make it possible to make a reference a git repo URL? With something like that, Aspire could clone and then either discover or be explicitly told what type of project is within for it to manage?

Because I don't think aspire should manage git repos. Pulling, updating, picking the right branch etc etc. That should be managed externally.

This would be a game changer, particularly for onboarding scenarios. I've followed a pattern of having a "home" repo that houses a docker-compose.yml definition that presumes the presence of other repositories in the past. Aspire is in a great position to improve on such a concept.

Right, I think it should be possible to have an aspire app host reference source anywhere but it shouldn't be responsible for the cloning and updating of the git repo.

[atrauzzi]

Yeah, I get what you're saying, it even makes sense. But I guess then it takes away some of that snappy convenience of being able to obtain the Aspire project itself and have things self-assemble.

If something like this were to exist, I don't think I'd want to see aspire go as far as branch management. Just clone and then put the developer back in the drivers seat, that's all.

[davidfowl]

If something like this were to exist, I don't think I'd want to see aspire go as far as branch management. Just clone and then put the developer back in the drivers seat, that's all.

That feels very incomplete IMO. I'm not sure we would want to ship a feature like that. I'm sure it could be built on top of our existing primitives, so if that's something interesting, somebody should build that and put up a nuget package.

[atrauzzi]

Oh! So that is to say, there's support for different source types to be added? And I guess then the rest of Aspire would (hopefully) pick up, assuming all went well?

[DamianEdwards]

The hosting primitives for DistributedApplicationBuilder are extensible, as long as any added resource eventually boils down to a container or an executable. So you could add overloads of AddProject() that accept git URLs or create a new AddGitProject() method that adds all the cloning logic but ultimately just adds normal project resources to the model.

[atrauzzi]

Love it. Something like .AddGithubProject("someoneororg/blabla") and then it would go cd ../../; mkdir "someoneororg/blabla"; cd "someoneororg/blabla"; git clone... (oversimplified!), and then do the regular add.

Although I agree it has an out-of-scopey flavour, I could see it being a very popular feature or extension. We'll see 🙂