RFC: Plugins support in Amplify Codegen

[phani-srikar]

This is a Request For Comments (RFC). RFCs are intended to elicit feedback regarding a proposed change to the Amplify Framework. Please feel free to post comments or questions here.

Summary:

The current Amplify GraphQL code generator needs improvements to make it more flexible and easily extensible. This RFC describes a plugins based architecture together with a re-designed configuration which could make the Amplify GraphQL code generator more flexible by allowing customizations at plugin level and easily extensible by supporting custom code generator plugins.

Motivation:

The motivation for this proposal comes from several developer requests listed in the Related Issues below. To summarize, developers want :

• To Generate type annotations at multiple destinations from a shared GraphQL schema
• To Generate type annotations or GraphQL statements from GraphQL schema split across multiple files
• Ability to use custom plugins to generate type annotations and also specify customizations for the plugins we provide.
• Better documentation for the codegen configuration

A plugins based architecture also brings several advantages to the current state of Amplify GraphQL code generator:

• More engagement from the developer community by encouraging them to build custom plugins. This could help us build a broader collection of plugins supporting multiple language targets quickly.
• Opportunity to refactor our current GraphQL code generation packages (statements, types and models generators) into smaller plugins specific to the target language. For example, @aws-amplify/graphql-typescript-types-generator , @aws-amplify/graphql-swift-types-generator etc.
• It would make releasing experimental or optional features easier because the change radius would be limited to a plugin. This would give us an opportunity to add more customizations for each codegen language target.

In the absence of a plugins based architecture, it would become increasingly difficult to maintain the current GraphQL code generation packages as the number of supported language targets increases with time.
It would also be harder to add customizations for each language target being code generated.

Detailed Design:

1. Re-Designed Configuration File: This is a sample configuration file that supports plugins based architecture:

schema:./amplify/backend/api/apiName/build/schema.graphql // resolved GraphQL schema. Accepts a list.
generates:
   statements:
      src/graphql:
        plugins:
            - @aws-amplify/graphql-typescript-docs-generator
    types: 
      src/API.ts:
        plugins:
            - @aws-amplify/graphql-typescript-types-generator
    models:
      src/models: // treats models generation for Amplify Datastore as another types generation task
        schema: ./amplify/backend/api/apiName/schema.graphql //overrides the top level schema
        plugins: 
            - @aws-amplify/appsync-typscript-modelgen-plugin

This is subject to change and will be updated in the RFC if need be. Any suggestions are welcome.

2. Plugin Interfaces: We will define the interfaces which the plugins are expected to implement in order for the generator to use them. The current types, statements and models generators will implement these interfaces and hence, can readily be used as plugins to the new architecture.

Some anticipated additions include:

• amplify-codegen-core package will be added which handles:
  • reading and parsing the re-designed configuration file
  • executing the specified plugins with appropriate plugin configurations
  • Collect and write the plugin outputs
• amplify-codegen-plugins-common package will be added which:
  • defines the interfaces a plugin must/can implement
  • defines the type definitions like PluginOutput and methods common to all plugins

Developer Experience:

The current GraphQL configuration (named .graphqlconfig.yml) is not well documented and its structure is confusing to developers. As part of this effort, we would like to make the configuration file as the source of truth for the generator. The amplify codegen add CLI walkthrough would help the developer create a minimal configuration file, which we will then prompt him/her to edit to add any customizations. We will clearly document how to create and maintain the configuration file.

The codegen add workflow will generate the re-designed configuration which can be re-configured using codegen configure.
The codegen statements workflow will use the plugins defined in the statements section of the re-designed config to generate the GraphQL operations.
The codegen types workflow will use the plugins defined in the types section of the re-designed config to generate the type annotations at corresponding output location. By default, types generators use the statements generation output. This behavior can however be customized to specify location to custom statements.
The codegen models workflow will use the plugins defined in the models section of the re-designed config to generate the type annotations at corresponding output location. These are intended to be used in combination with Amplify Datastore.
The codegen workflow will run the statements, types and models workflows.

Refer Appendix for more information.

We will provide a guide to creating custom plugins for the developers. We will also provide API documentation for the supported plugins including the customizations they support.

Drawbacks and Adoption Strategy:

The developer would need to migrate to using the re-designed configuration.
We will continue to support the options that are present in the current configuration, for example, maxDepth, region, apiId etc. The output of the generators should remain the same when appropriately configured.
In addition to the detailed documentation for the re-designed configuration, we will provide a migration guide to help the developers quickly make the switch.
The initial release will provide a Feature Flag with a pre-defined deprecation date which allows the existing customers to test the plugins architecture before making the switch.

Related Issues:

• [codegen] graphql config - multiple extension outputs from a project amplify-codegen#49
• How do I generate API from my .graphql files without generating entire api? amplify-codegen#39 (comment)
• How do I generate exportable graphql input types using amplify codegen? amplify-codegen#44
• RFC: Amplify codegen generating enums.js file in src/graphql folder amplify-codegen#35

Appendix:

Architecture Overview

This section gives an overview of the re-designed architecture. It is subject to change and should be treated as a sample.

Consider a simple schema using a statements generator, two types generators and a models generator plugins:

schema:./amplify/backend/api/apiName/build/schema.graphql // resolved GraphQL schema
generates:
   statements:
      src/graphql:
        plugins:
            - <statements generator plugin>
    types: 
      src/API.ts:
        plugins:
            - <types generator plugin 1>
      src/other/API.ts: 
        schema: ./amplify/backend/api/apiName/schema.graphql //overrides the top level schema
        plugins: 
            - <types generator plugin 2>
    models:
      src/models:
        schema: ./amplify/backend/api/apiName/schema.graphql //overrides the top level schema
        plugins:
            - <models generator plugin>

amplify codegen statements workflow: Generates GraphQL statements from a given GraphQL schema.

amplify codegen types workflow: Generates type annotations in specific target language from given GraphQL schema and GraphQL statements.

amplify codegen models workflow: Generates type annotations in specific target language from given GraphQL schema, to be used with Amplify Datastore

amplify codegen workflow: Generates GraphQL statements from a given GraphQL schema. Then, uses the schema and the generated statements to generate type annotations in specific target language. Also, generates the type annotations called Datastore Models from the given GraphQL schema.

The context being passed to various plugins will contain the information that it needs to generate the code.

Terminology:

• “GraphQL statements“ refers to the GraphQL queries, mutations and subscriptions.
• “statements generator” refers to a plugin that generates GraphQL statements from given GraphQL schema. Also referred to as “docs generator”. See Example.
• “types generator” refers to a plugin that generates language specific type annotations from given GraphQL schema and GraphQL statements. See Example.
• “models generation” refers generating the language specific type annotations or Models that are used to work with Amplify Datastore. Refer for more information.
• “codegen”, “typegen”, “docgen” are short forms for “code generation”, “types generator” and “statements generator” respectively.
• “type annotations” refers to the language specific type definitions generated for @model annotated types and statements. For example,

// Given the schema with a simple query:
    type Todo {
      id: ID!
      name: String!
      description: String
      createdAt: AWSDateTime!
      updatedAt: AWSDateTime!
    }
    
    type Query {
      getTodo(id: ID!): Todo
    }
    
    // Type annotations in typescript look like:
    export type Todo = {
      __typename: "Todo",
      id?: string,
      name?: string,
      description?: string | null,
      createdAt?: string,
      updatedAt?: string,
    };
    
    export type GetTodoQueryVariables = {
      id?: string,
    };
    
    export type GetTodoQuery = {
      getTodo?:  {
        __typename: "Todo",
        id: string,
        name: string,
        description?: string | null,
        createdAt: string,
        updatedAt: string,
      } | null,
    };

References:

The current code generation packages (statements, types and models) can be found here

Special thanks to pointing out the following resources in some of the feature-requests:

• https://github.com/apollographql/apollo-tooling
• https://github.com/dotansimha/graphql-code-generator

[nubpro]

Will this RFC potentially resolve aws-amplify/amplify-codegen#474?
Basically we want to be able to use GraphQL docs generated from the schema and use them directly in our lambda functions.
Currently, we have to copy the queries and move them into the function folder manually, and it is becoming a very tedious task whenever the schema needed a change. I can't imagine how developers can easily maintain this if the number of functions grow exponentially.

Here are the methods I've tried:

1. Build options - which is by copying the graphql folder into the functions whenever you run amplify push. This wont work properly because the docs are generated after the functions are packaged not before. Which made me think we can force codegen in the script but there's another problem with the CLI ([request?] amplify codegen does not reflect updated schema from local copy amplify-codegen#113 (comment)). And also, this doesn't scale and very unoptimized since we're dumping everything into the function directory even though it only accesses a single model.
2. Writing a plugin, honestly I dont even know where to start and is it even worth our time to spend digging on how the CLI works. You can't attach to any of the events because of the issue I listed in number 1. Hence, I was thinking perhaps I could use a command that would scan through all the functions and see which AppSync API that they have been given permission to, and with that information, I can generate the selected graphql docs for the function to consume. The main blocker of this is that, there's very little to no information as how I can do this - how do I go through all the functions?, how do I see what is configured for the function? I could perhaps study the CLI source codes and see how existing plugins are built but that's a lot of effort and I'm probably going to end up pulling my own hair.

I honestly think there's a lot of improvement to be made if you want developers to start taking amplify plugins seriously. The assumption that we know most of the things you guys know needs to be off the table.

[phani-srikar]

Hi @nubpro. This RFC would help you generate docs at multiple destinations by specifying multiple docs generation tasks. This will save you the hassle to copy over the generated docs to your lambda function. Thank you for your feedback.

[n10000k]

I support this RFC. Would make life a lot easier having the ability to dish out graphql files to separate locations. Example being many lambda functions.

[asmajlovicmars]

This is perfect!

Thought it would be a smaller change, but seems like a much bigger, and a better thing. Btw, will cdk be able to invoke the "docgen", and generate graphql files based on the provided schema? Thanks @phani-srikar and the team for this very detailed proposal!

[joekiller]

Hello I support the RFC as it would be useful to codegen queries into multiple functions of various languages. IE we have a lot of nodejs but need to support python as well. Codegen queries and models into both of those directories would be very useful. Also codegen into a lambda layer etc.

[camin-mccluskey]

I would also like to register my support for this RFC. As Amplify projects grow it is almost inevitable that other systems (particularly Lambdas) will need to touch the provisioned datastores. Having code generated models for these systems would be incredibly helpful

[sameerdewan]

Where is this effort currently? This is something needed. Client -> GraphQL interaction is great out of the box, but more server side logic driven applications require graphql on the backend, and for that matter, potentially over multiple functions.

[dsharygin]

+1. We have dozens of functions doing asynchronous processing in response to DynamoDB streams. Having to copy/paste the queries into each function inevitably leads to schema mismatches and bugs.

[chrisl777]

We have a monorepo set up with multiple React front-end projects, along with several Lambda functions.

• We need to use the GraphQL API in all of these projects and it is cumbersome to copy codegen changes to all folders.
• We're using create-react-app, which out of the box does not let you import files from outside of the root of each front-end project.
• It is tedious and error-prone to copy/paste the generated API files between projects within the monorepo.

[redjonzaci]

We have Lambda functions in Go and we also want to reuse the generated GraphQL code.

[tjrivera]

We have a monorepo set up with multiple React front-end projects, along with several Lambda functions.

• We need to use the GraphQL API in all of these projects and it is cumbersome to copy codegen changes to all folders.
• We're using create-react-app, which out of the box does not let you import files from outside of the root of each front-end project.
• It is tedious and error-prone to copy/paste the generated API files between projects within the monorepo.

Hey @chrisl777, I feel your pain on this one. Our team ended up managing all of our Amplify related resources in a package within our monorepo, other apps (we have a react-native app, some data processes, web app, etc.) import those resources from the data package. This makes it easier for us to manage schema updates and code generation in one place and import them in other projects from our package.

Sounds like this doesn't quite fit with your current CRA pattern, but might encourage you to check out some monorepo tooling (nx, lerna, yarn workspaces, turborepo, etc.)

[oe-bayram]

Will this RFC also resolve aws-amplify/amplify-codegen#251?

A more versatile approach could involve defining the depth of connection paths for specific queries.

For instance, in the configuration file, we could stipulate the depth for necessary connections for a query like listInvoices as follows:

listInvoices:
    order -> customer -> address
    order -> positions -> product

If the maxDepth is set to 2, the plugin could generate the query considering this depth and also take into account the paths defined in the configuration file. This would allow for greater flexibility and control over query depth on a per-query basis.

[johnemcbride]

Thumbs up on this too - it's a bit odd I have to copy and paste code around to write my lambda functions.

[hanna-becker]

I stumbled across the following limitation today and while searching through Github issues was redirected here:

When using multiple frontends with one Amplify backend as described in the docs here, when selecting 'no' for ? Do you plan on modifying this backend? for the second frontend, we can't perform amplify codegen there, as it doesn't have a schema.json/schema.graphql file to generate client code from. This is super annoying, because of course we want to use codegen in our other client, we just don't want developers to be able to modify the backend from that repo. I'm not sure this RFC covers this use case, but it would be super awesome if amplify pull in this "read-only mode" had the option to pull the schema.json file so that codegen works.