Kiota.Config for managing multiple API dependencies in a project.

[darrelmiller]

Context

Kiota generates client code for an API and stores parameters in a kiota.lock file. A project can contain multiple API clients, but they are independently managed. Kiota has no awareness that an app has a dependency on multiple APIs, even though that is a core use case.

Current Challenges 

• Client code generation is not reproducible if API description changes
• Kiota doesn’t have a good solution for APIs that use multiple security schemes.
• Kiota doesn’t provide any support for generating auth providers with the required permissions, partially because currently we generate one client for APIs that use different schemes. How would we know which auth provider to generate.
• Kiota doesn’t have a good story for acquiring a client identifier. e.g. apikey or OAuth2 ClientId. This could be possible if the OpenIDConnect URL pointed to a dynamic registration endpoint.
• If an application has multiple kiota clients, there is currently no way perform operations that correspond to all of the clients.

We have previously described Kiota's approach to managing API dependencies as consistent with the way people manage packages in a project. However, currently our tooling doesn't behave that way. We treat each dependency independently.

Proposal

We should introduce a new Kiota.config file that holds the input parameters required to generate the API Client code. Currently kiota.lock is used to capture what the parameters were at the time of generation and can be used to regenerate based on the parameters in the file. This creates a mixture of purposes for the file.

We did consider creating one kiota.config file as as a peer of the language project file, however, For someone who wants to generate multiple clients for an API in different languages, this would be a bit annoying. An alternative would be to allow the kiota.config file to move further up the folder structure and support generation in multiple languages from a single file. This is more consistent with what TypeSpec are doing and would be helpful for generating CLI and docs as well as a library.

Here is an example of what the kiota.config file could look like.

{
  "name": "My application",
  "apis": {
    "graphDelegated": {
      "descriptionHash": "9EDF8506CB74FE4414676494FCFCE4681EBCA47D386F845BDDF238413B22C6A82D47DAC9E144A431FA0C9AF6126027F2E58619707FC1A00DED16BCC9AE53D6A3",
      "descriptionLocation": "https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/master/openapi/v1.0/openapi.yaml",
      "includePatterns": [
        "/me/chats#GET",
        "/me#GET"
      ],
      "excludePatterns": [],
      "baseUrl": "https://graph.microsoft.com/v1.0/", 
      "output": [
        {
          "language": "csharp",
          "outputPath": "./generated/",
          "clientClassName": "MicrosoftGraphClient",
          "clientNamespaceName": "MicrosoftGraph",
          "features": {
            "authenticationProvider": "Microsoft.Kiota.Authentication.AzureAuthProvider",
            "authenticationParameters"{
              "clientId": "guid"
            },
            "structuredMediaTypes": {
              "application/json": {
                "serializer": "Microsoft.Kiota.Serialization.Json.JsonSerializationWriterFactory",
                "deserializer": "Microsoft.Kiota.Serialization.Json.JsonParseNodeFactory"
              }
            },
            "usesBackingStore": false,
            "includeAdditionalData": false
          }
        }
      ],
      "disabledValidationRules": []
    }
  }
}

Note that in this example we added suggestions for new parameters related to authentication. If we are to improve the generation experience so that we read the security schemes information from the OpenAPI, then we will need to have some place to configure what providers we will use for those schemes.

The API Manifest file can be used as a replacement for the kiota.lock file as a place to capture a snapshot of what information was used to perform code generation and what APIs that gives the application access to.

Principles

• Kiota.config can be edited by hand, or use the Kiota tooling to edit it. The client generation process does not write anything into Kiota.config. It is an input to the process only.
• kiota.apimanifest is created during the process of client code generation and is a snapshot of the parameters used to drive the generation process. It should not be edited by hand.

Tooling commands to manage Kiota.config

Command	Example	Description
init	kiota init --name <appName>	Creates a kiota.config file
add api	kiota add api --name <apiname> --openapi <urlToApiDescription>	Adds an entry for an API with passed parameters and default values
add output	kiota add output --name MyApi --lang python --outputPath ./pythonClient	Adds information about a new output artifact that should be generated
generate	kiota generate	Outputs kiota.apimanifest and source for each of the output objects

In the past we have had both a generate and an update comment. This is because if it was the first time running, generate would set defaults for all the properties that were not set on the command line. However, now that we have add output, it can be used to set the defaults and generate can just read from the kiota.config file. Question 1: Do we keep the parameters in generate to allow you to update kiota.config and maintain some backward compat with previous commands?

Scenarios using the command line tool

Get started to generate an API

kiota init --name Myapp
kiota add api --name MyApi --openapi https://example.org/api/openapi.json   // Can we add using -k ?
kiota add output --name MyApi --lang csharp --outputPath ./csharpClient
kiota generate

Add a second language to generate an API

kiota add output --name MyApi --lang python --outputPath ./pythonClient
kiota generate --name MyApi --lang python  // Generate just the Python client for MyApi

Remove a language

kiota remove output --name MyApi --lang python 
                                                                                

Question 2: Does removing the output remove the generated code? The entire folder?
Question 3: How do we uniquely identify an Output? Should we be able to generate two Python clients for MyApi?

Remove an API

kiota remove api --name MyApi                           

Question 5: Does removing the output remove the generated code? The entire folder?

JSON Schema for Kiota.Config

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "apis": {
      "type": "object",
      "patternProperties": {
        ".*": {
          "type": "object",
          "properties": {
            "descriptionLocation": {
              "type": "string"
            },
            "descriptionHash": {
              "type": "string"
            }
          },
          "descriptionHash": {
            "type": "string"
          },
          "descriptionLocation": {
            "type": "string"
          },
          "includePatterns": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "excludePatterns": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "baseUrl": {
            "type": "string"
          },
          "output": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "language": {
                  "type": "string"
                },
                "outputPath": {
                  "type": "string"
                },
                "clientClassName": {
                  "type": "string"
                },
                "clientNamespaceName": {
                  "type": "string"
                },
                "features": {
                  "type": "object",
                  "properties": {
                    "authenticationProvider": {
                      "type": "string"
                    },
                    "authenticationParameters": {
                      "type": "object"
                    }
                  },
                  "structuredMediaTypes": {
                    "type": "object",
                    "patternProperties": {
                      ".*": {
                        "type": "object",
                        "properties": {
                          "serializer": {
                            "type": "string"
                          },
                          "deserializer": {
                            "type": "string"
                          }
                        }
                      }
                    }
                  },
                  "usesBackingStore": {
                    "type": "boolean"
                  },
                  "includeAdditionalData": {
                    "type": "boolean"
                  }
                }
              }
            }
          }
        },
        "disabledValidationRules": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    }
  }
}

[baywet]

Thanks for starting to write this down.
I believe that having two add commands is a lot of ceremony to get people to call their apis. Maybe we could design things with a single command and dedupe existing entries?
A1: If we start removing parameters, we'll need to major bump. We should consider that aspect in the release cadence. If we do remove the parameters, then I don't think we need an update command anymore. Just generate with --update should be sufficient.
A2: Why not add a switch to let the user control the removal of the folder like --delete-code or something?
A3: I can see scenarios where a big repo has multiple clients to the same API in the same language (different feature boundaries). IMHO the divide between API and output might be overkill.
A4: Counting is hard ;-)
A5: same as A3?

[baywet]

@andreaTP that's the last thing we were referring to in the call earlier

[darrelmiller]

@baywet
A1: I'm fine with starting with the assumption that we maintain compatibility, however if it gets too complex, we should consider rethinking the experience completely with the new model.
A2: That's fine. Now the question is what should the default be. To keep it, or to leave it. My gut tells me that we should try and keep things clean.
A3: I think this is consistent with our thinking around having different clients for security schemes and API versions.
A4: Yes it is.
A5: Ack.

I will try and take another swing at proposing commands that don't require two commands to add an API.

[andreaTP]

I have skimmed through this proposal and I like it very much!
Thanks for always thinking ahead!

A few quick notes:

1. I love the proposal but I believe that the Config design should be a natural consequence other than the motivating principle behind those changes, the overall story of handling multiple APIs probably already has quite some invariants.
2. As much as possible this should be designed and implemented as a single command (or, at least, having the option to use only a single one), as soon as we resolve the issues with idempotency, we(users 🙂) should be free to use a single command to generate the full output, and it needs to be 100% reproducible.
3. I'm not sure about the correct design for handling the matrix of shared Http clients + shared Adapters(and multiple auth mechanisms), but, since we are looking at it, let's take into account how we can inject the HttpClient e.g. down to the AccessTokenProvider ( we should be able to not instantiate an extra client like I do here)

[pschaeflein]

Not wild about add output ...

The "thing" I'm adding to the config is...:

• a client (add client ...)?
• code (add code ...)?
• Or, lean into the name (add egg ...) 😆

I do like the idea of separate configuration/lock files.

[sebastienlevert]

I think client make a lot of sense and plays nicely with the naming convention we are generating anyways! Thanks for the feedback @pschaeflein

[samcragg]

Really exciting feature, was about to propose something similar!

I notice this issue has been marked as closed and it's in the 1.12 release notes, however, there doesn't appear to be any documentation on it and it looks like it's behind a feature flag.

Is this feature ready or is there more work to be done? I'm happy to help volunteer my time, for example, by having a go at adding the docs (once I have a quick play with it) and I assume I just add them over here?

[baywet]

Hey @samcragg
Glad to hear this feature is interesting to you!
We're still working out our plans so far, they are not finalized, but here is the general idea:

1. have more people try out this new feature and provide us feedback around the developer experience (please go ahead)
2. put out bundle packages to simplify the registration of serialization provider packages (gap with this new experience)
3. finalize the plugins story (lots of moving parts today, and this is a high priority for us)
4. update the doc
5. clean up all the things behind --ebc flag
6. remove the older rundundant commands (kiota add, kiota update, ...), and the feature flag
7. major rev

If you want to help on the docs, you can get started by submitting a PR to document the new commands and clean up the old ones. It'd be much appreciated!