Allow mapping directly to a type but not using its standard JSON serialization #38

benjaminjkraft · 2021-04-22T18:11:30Z

You might want to map your GraphQL DateTime to time.Time, but you might want to only include microseconds (because Python doesn't like them). You can do that by defining your own type, but then you have to cast at every call, which is the kind of boilerplate genqlient is designed to reduce! We could instead let you just define marshal/unmarshal functions, like gqlgen does.

The text was updated successfully, but these errors were encountered:

Summary: We had this setting called "scalars", which said: bind this GraphQL type to this Go type, rather than the one you would normally use. It's called that because it's most useful for custom scalars, where "the one you would normally use" is "error: unknown scalar". But nothing ever stopped you from using it for a non-scalar type. I was planning on removing this functionality, because it's sort of a rough edge, but a discussion with Craig found some good use cases, so instead, in this commit, I document it better and add some slightly nicer ways to specify it. Specifically, here are a few potential non-scalar use cases: - bind a GraphQL enum to a nonstandard type (or even `string`) - bind an input type to some type that has exactly the fields you want; this acts as a sort of workaround for issues #14 and #44 - bind an object type to your own struct, so as to add methods to it (this is the use case Craig raised) - bind an object type to your own struct, so as to share it between multiple queries (I believe named fragments will address this case better, but it doesn't hurt to have options) - bind a GraphQL list type to a non-slice type in Go (presumably one with an UnmarshalJSON method), or any other different structure The latter three cases still have the sharp edge I was originally worried about, which is that nothing guarantees that the fields you request in the query are the ones the type expects to get. But I think it's worth having the option, with appropriate disclaimers. The main change to help support that better is that you can now specify the type inline in the query, as an alternative to specifying it in the config file; this means you might map a given object to a given struct, but only in some cases, and when you do you have a chance to look at the list of fields you're requesting. Additionally, I renamed the config field from "scalars" to "bindings" (but mentioned it in a few places where you might go looking for how to map scalars, most importantly the error message you get for an unknown (custom) scalar). While I was making a breaking change, I also changed it to be a `map[string]<struct>` instead of a `map[string]string`, because I expect to add more fields soon, e.g. to handle issue #38. Finally, since the feature is now intended/documented, I added some tests, although it's honestly quite simple on the genqlient side. Test Plan: make tesc Reviewers: csilvers, marksandstrom, adam, miguel

…pe (#69) ## Summary: We had this setting called "scalars", which said: bind this GraphQL type to this Go type, rather than the one you would normally use. It's called that because it's most useful for custom scalars, where "the one you would normally use" is "error: unknown scalar". But nothing ever stopped you from using it for a non-scalar type. I was planning on removing this functionality, because it's sort of a rough edge, but a discussion with Craig found some good use cases, so instead, in this commit, I document it better and add some slightly nicer ways to specify it. Specifically, here are a few potential non-scalar use cases: - bind a GraphQL enum to a nonstandard type (or even `string`) - bind an input type to some type that has exactly the fields you want; this acts as a sort of workaround for issues #14 and #44 - bind an object type to your own struct, so as to add methods to it (this is the use case Craig raised) - bind an object type to your own struct, so as to share it between multiple queries (I believe named fragments will address this case better, but it doesn't hurt to have options) - bind a GraphQL list type to a non-slice type in Go (presumably one with an UnmarshalJSON method), or any other different structure The latter three cases still have the sharp edge I was originally worried about, which is that nothing guarantees that the fields you request in the query are the ones the type expects to get. But I think it's worth having the option, with appropriate disclaimers. The main change to help support that better is that you can now specify the type inline in the query, as an alternative to specifying it in the config file; this means you might map a given object to a given struct, but only in some cases, and when you do you have a chance to look at the list of fields you're requesting. Additionally, I renamed the config field from "scalars" to "bindings" (but mentioned it in a few places where you might go looking for how to map scalars, most importantly the error message you get for an unknown (custom) scalar). While I was making a breaking change, I also changed it to be a `map[string]<struct>` instead of a `map[string]string`, because I expect to add more fields soon, e.g. to handle issue #38. Finally, since the feature is now intended/documented, I added some tests, although it's honestly quite simple on the genqlient side. ## Test plan: make tesc Author: benjaminjkraft Reviewers: csilvers, aberkan, dnerdy, MiguelCastillo Required Reviewers: Approved by: csilvers Checks: ⌛ Test (1.17), ⌛ Test (1.16), ⌛ Test (1.15), ⌛ Test (1.14), ⌛ Test (1.13), ⌛ Lint, ⌛ Test (1.17), ⌛ Test (1.16), ⌛ Test (1.15), ⌛ Test (1.14), ⌛ Test (1.13), ⌛ Lint Pull request URL: #69

genqlient has some code (`imports.go`) dedicated to tracking which imports we need and avoiding conflicts, as well as converting a (restricted) Go expression like `map[string]github.com/me/mypkg.MyType` to an import (`github.com/me/mypkg`) and a type-reference (`map[string]mypkg.MyType`) to be used in the context of that import, and at least making some attempt to track conflicts. (Right now the conflict-avoidance is not very smart, and not very well tested, but it comes up rarely anyway.) Sadly, that code was a bit cumbersome to use, because you had to first register the imports (typically from `convert.go`), then use them (often from the template). In this commit I refactor the order we write things in order to allow a significant simplification of how we import; in particular we no longer have to guess in advance what imports which template will need; it can just do `{{ref <expr>}}` as before, and it just works. To do this, I: - changed the importer to have only one API, which adds the import if needed, and returns the reference either way - added a check that we don't add imports after they're written - reorganized the toplevel templates a bit to make sure that check never fires; we now generate all the types and operations, then write the imports and glue it all together This removes a bunch of silly code, and should simplify the process of adding custom (un)marshalers (#38). While I was at it, I put the documentation of what expressions we support in a more visible place, and added a type-assertion that your custom context type implements context.Context (if applicable). Test plan: make check Reviewers: marksandstrom, mahtab, jvoll, adam, miguel, steve

## Summary: genqlient has some code (`imports.go`) dedicated to tracking which imports we need and avoiding conflicts, as well as converting a (restricted) Go expression like `map[string]github.com/me/mypkg.MyType` to an import (`github.com/me/mypkg`) and a type-reference (`map[string]mypkg.MyType`) to be used in the context of that import, and at least making some attempt to track conflicts. (Right now the conflict-avoidance is not very smart, and not very well tested, but it comes up rarely anyway.) Sadly, that code was a bit cumbersome to use, because you had to first register the imports (typically from `convert.go`), then use them (often from the template). In this commit I refactor the order we write things in order to allow a significant simplification of how we import; in particular we no longer have to guess in advance what imports which template will need; it can just do `{{ref <expr>}}` as before, and it just works. To do this, I: - changed the importer to have only one API, which adds the import if needed, and returns the reference either way - added a check that we don't add imports after they're written - reorganized the toplevel templates a bit to make sure that check never fires; we now generate all the types and operations, then write the imports and glue it all together This removes a bunch of silly code, and should simplify the process of adding custom (un)marshalers (#38). While I was at it, I put the documentation of what expressions we support in a more visible place, and added a type-assertion that your custom context type implements context.Context (if applicable). ## Test plan: make check Author: benjaminjkraft Reviewers: StevenACoffman, dnerdy, aberkan, jvoll, mahtabsabet, MiguelCastillo Required Reviewers: Approved By: StevenACoffman, dnerdy Checks: ⌛ Test (1.17), ⌛ Test (1.16), ⌛ Test (1.15), ⌛ Test (1.14), ⌛ Lint, ⌛ Test (1.17), ⌛ Test (1.16), ⌛ Test (1.15), ⌛ Test (1.14), ⌛ Lint Pull Request URL: #101

In this commit I refactor the argument-generation logic to move most of the code out of the template and into the type-generator. This logic predates #51, and I didn't think to update it there, but I think it benefits from similar treatment, for similar reasons. Specifically, the main change is to treat variables as another struct type we can generate, rather than handling them inline as a `map[string]interface{}`. Users still pass them the same way, but instead of putting them into a `map[string]interface{}` and JSONifying that, we generate a struct and put them there. This turns out to simplify things quite a lot, because we already have a lot of code to generate types. Notably, the omitempty code goes from a dozen lines to basically two, and fixes a bug (#43) in the process, because now that we have a struct, `json.Marshal` will do our work for us! (And, once we have syntax for it (#14), we'll be able to handle field-level omitempty basically for free.) More importantly, it will simplify custom marshalers (#38, forthcoming) significantly, since we do all that logic at the containing-struct level, but will need to apply it to arguments. It does require one breaking change, for folks implementing the `graphql.Client` API (rather than just calling `NewClient`): we now pass them variables as an `interface{}` rather than a `map[string]interface{}`. For most callers, including Khan/webapp, this is basically a one-line change to the signature of their `MakeRequest`. Issue: #38 Issue: #43 Test plan: make check Reviewers: marksandstrom, steve, jvoll, mahtab, adam, miguel

This is useful if you want to bind to a type you don't control (or use for other things) but need different serialization than its default. This is a feature gqlgen has and we've found it very useful. For example, in webapp we want to bind `DateTime` to `time.Time`, but its default serialization is not compatible with Python, so currently we have to bind to a wrapper type and cast all over the place, which is exactly the sort of boilerplate genqlient is supposed to avoid. For unmarshaling, the implementation basically just follows the existing support for abstract types; instead of calling our own generated helper, we now call your specified function. This required some refactoring to abstract the handling of custom unmarshalers generally from abstract types specifically, and to wire in not only the unmarshaler-name but also the `generator` (in order to compute the right import alias). For marshaling, I had to implement all that stuff over again; it's mostly parallel to unmarshaling (and I made a few minor changes to unmarshaling to make the two more parallel). Luckily, after #103 I at least only had to do it once, rather than implementing the same functionality for arguments and for input-type fields. It was still quite a bit of code; I didn't try to be quite as completionist about the tests as with unmarshal but still had to add a few. Issue: #38 Test plan: make check Reviewers: marksandstrom, steve, adam, jvoll, miguel, mahtab

In this commit I refactor the argument-generation logic to move most of the code out of the template and into the type-generator. This logic predates #51, and I didn't think to update it there, but I think it benefits from similar treatment, for similar reasons. Specifically, the main change is to treat variables as another struct type we can generate, rather than handling them inline as a `map[string]interface{}`. Users still pass them the same way, but instead of putting them into a `map[string]interface{}` and JSONifying that, we generate a struct and put them there. This turns out to simplify things quite a lot, because we already have a lot of code to generate types. Notably, the omitempty code goes from a dozen lines to basically two, and fixes a bug (#43) in the process, because now that we have a struct, `json.Marshal` will do our work for us! (And, once we have syntax for it (#14), we'll be able to handle field-level omitempty basically for free.) More importantly, it will simplify custom marshalers (#38, forthcoming) significantly, since we do all that logic at the containing-struct level, but will need to apply it to arguments. It does require one breaking change, for folks implementing the `graphql.Client` API (rather than just calling `NewClient`): we now pass them variables as an `interface{}` rather than a `map[string]interface{}`. For most callers, including Khan/webapp, this is basically a one-line change to the signature of their `MakeRequest`. Issue: #38 Issue: #43 Test plan: make check Reviewers: marksandstrom, steve, jvoll, mahtab, adam, miguel

This is useful if you want to bind to a type you don't control (or use for other things) but need different serialization than its default. This is a feature gqlgen has and we've found it very useful. For example, in webapp we want to bind `DateTime` to `time.Time`, but its default serialization is not compatible with Python, so currently we have to bind to a wrapper type and cast all over the place, which is exactly the sort of boilerplate genqlient is supposed to avoid. For unmarshaling, the implementation basically just follows the existing support for abstract types; instead of calling our own generated helper, we now call your specified function. This required some refactoring to abstract the handling of custom unmarshalers generally from abstract types specifically, and to wire in not only the unmarshaler-name but also the `generator` (in order to compute the right import alias). For marshaling, I had to implement all that stuff over again; it's mostly parallel to unmarshaling (and I made a few minor changes to unmarshaling to make the two more parallel). Luckily, after #103 I at least only had to do it once, rather than implementing the same functionality for arguments and for input-type fields. It was still quite a bit of code; I didn't try to be quite as completionist about the tests as with unmarshal but still had to add a few. Issue: #38 Test plan: make check Reviewers: marksandstrom, steve, adam, jvoll, miguel, mahtab

## Summary: In this commit I refactor the argument-generation logic to move most of the code out of the template and into the type-generator. This logic predates #51, and I didn't think to update it there, but I think it benefits from similar treatment, for similar reasons. Specifically, the main change is to treat variables as another struct type we can generate, rather than handling them inline as a `map[string]interface{}`. Users still pass them the same way, but instead of putting them into a `map[string]interface{}` and JSONifying that, we generate a struct and put them there. This turns out to simplify things quite a lot, because we already have a lot of code to generate types. Notably, the omitempty code goes from a dozen lines to basically two, and fixes a bug (#43) in the process, because now that we have a struct, `json.Marshal` will do our work for us! (And, once we have syntax for it (#14), we'll be able to handle field-level omitempty basically for free.) More importantly, it will simplify custom marshalers (#38, forthcoming) significantly, since we do all that logic at the containing-struct level, but will need to apply it to arguments. It does require two breaking changes: 1. For folks implementing the `graphql.Client` API (rather than just calling `NewClient`): we now pass them variables as an `interface{}` rather than a `map[string]interface{}`. For most callers, including Khan/webapp, this is basically a one-line change to the signature of their `MakeRequest`, and it should be a lot more future-proof. 2. genqlient's handling of the `omitempty` option has changed to match that of `encoding/json`, in particular it now never considers structs "empty". The difference was never intentional (I just didn't realize that behavior of `encoding/json`); arguably our behavior was more useful but I think that's outweighed by the value of consistency with `encoding/json` as well as the simpler and more correct implementation (fixing #43 is actually quite nontrivial otherwise). Once we have custom unmarshaler support (#38), users will be able to map a zero value to JSON null if they wish, which is mostly if not entirely equivalent for GraphQL's purposes. Issue: #38 Issue: #43 ## Test plan: make check Author: benjaminjkraft Reviewers: StevenACoffman, dnerdy, aberkan, jvoll, mahtabsabet, MiguelCastillo Required Reviewers: Approved By: StevenACoffman, dnerdy Checks: ✅ Test (1.17), ✅ Test (1.16), ✅ Test (1.15), ✅ Test (1.14), ✅ Lint, ✅ Lint, ✅ Test (1.17), ✅ Test (1.16), ✅ Test (1.15), ✅ Test (1.14) Pull Request URL: #103

This is useful if you want to bind to a type you don't control (or use for other things) but need different serialization than its default. This is a feature gqlgen has and we've found it very useful. For example, in webapp we want to bind `DateTime` to `time.Time`, but its default serialization is not compatible with Python, so currently we have to bind to a wrapper type and cast all over the place, which is exactly the sort of boilerplate genqlient is supposed to avoid. For unmarshaling, the implementation basically just follows the existing support for abstract types; instead of calling our own generated helper, we now call your specified function. This required some refactoring to abstract the handling of custom unmarshalers generally from abstract types specifically, and to wire in not only the unmarshaler-name but also the `generator` (in order to compute the right import alias). For marshaling, I had to implement all that stuff over again; it's mostly parallel to unmarshaling (and I made a few minor changes to unmarshaling to make the two more parallel). Luckily, after #103 I at least only had to do it once, rather than implementing the same functionality for arguments and for input-type fields. It was still quite a bit of code; I didn't try to be quite as completionist about the tests as with unmarshal but still had to add a few. Issue: #38 Test plan: make check Reviewers: marksandstrom, steve, adam, jvoll, miguel, mahtab

## Summary: This is useful if you want to bind to a type you don't control (or use for other things) but need different serialization than its default. This is a feature gqlgen has and we've found it very useful. For example, in webapp we want to bind `DateTime` to `time.Time`, but its default serialization is not compatible with Python, so currently we have to bind to a wrapper type and cast all over the place, which is exactly the sort of boilerplate genqlient is supposed to avoid. For unmarshaling, the implementation basically just follows the existing support for abstract types; instead of calling our own generated helper, we now call your specified function. This required some refactoring to abstract the handling of custom unmarshalers generally from abstract types specifically, and to wire in not only the unmarshaler-name but also the `generator` (in order to compute the right import alias). For marshaling, I had to implement all that stuff over again; it's mostly parallel to unmarshaling (and I made a few minor changes to unmarshaling to make the two more parallel). Luckily, after #103 I at least only had to do it once, rather than implementing the same functionality for arguments and for input-type fields. It was still quite a bit of code; I didn't try to be quite as completionist about the tests as with unmarshal but still had to add a few. Issue: #38 ## Test plan: make check Author: benjaminjkraft Reviewers: StevenACoffman, dnerdy, benjaminjkraft, aberkan, jvoll, mahtabsabet, MiguelCastillo Required Reviewers: Approved By: StevenACoffman, dnerdy Checks: ✅ Test (1.17), ✅ Test (1.16), ✅ Test (1.15), ✅ Test (1.14), ✅ Lint, ✅ Test (1.17), ✅ Test (1.16), ✅ Test (1.15), ✅ Test (1.14), ✅ Lint Pull Request URL: #104

benjaminjkraft added this to the Khan milestone Apr 22, 2021

benjaminjkraft mentioned this issue Aug 27, 2021

Document and improve support for binding non-scalars to a specific type #69

Merged

benjaminjkraft self-assigned this Sep 9, 2021

benjaminjkraft added the enhancement New feature or request label Sep 11, 2021

benjaminjkraft mentioned this issue Sep 16, 2021

Refactor import-loading to simplify the type-generation code #101

Merged

benjaminjkraft mentioned this issue Sep 17, 2021

Refactor argument-handling to use a struct #103

Merged

benjaminjkraft mentioned this issue Sep 17, 2021

Add support for binding with a custom marshal/unmarshal function #104

Merged

benjaminjkraft linked a pull request Sep 17, 2021 that will close this issue

Add support for binding with a custom marshal/unmarshal function #104

Merged

benjaminjkraft closed this as completed in #104 Sep 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow mapping directly to a type but not using its standard JSON serialization #38

Allow mapping directly to a type but not using its standard JSON serialization #38

benjaminjkraft commented Apr 22, 2021

Allow mapping directly to a type but not using its standard JSON serialization #38

Allow mapping directly to a type but not using its standard JSON serialization #38

Comments

benjaminjkraft commented Apr 22, 2021