L70: @grpc/proto-loader TypeScript Type Generator CLI Tool #183

murgatroid99 · 2020-06-01T23:48:21Z

Discussion thread: https://groups.google.com/g/grpc-io/c/FwLprMC-5Qc

L70-node-proto-loader-type-generator.md

alexander-fenster · 2020-06-02T18:11:06Z

L70-node-proto-loader-type-generator.md

+ - `--oneofs`: Indicates that virtual "oneof" fields will be set to the present field's name in the output
+ - `--includeDirs=<directories...>`, `-I`: A list of directories to search for included `.proto` files
+
+ - `--outDir`, `-O`: The path in which to output the `.d.ts` file


if the output is a single .d.ts file, would it be better to have a regular -o option for an output file name, dumping it to stdout by default if the path is not given?

if it's not a single .d.ts but multiple .d.ts (e.g. one per service), it's worth mentioning that too.

I've clarified what files will be generated and in what directory structure.

DBosley · 2020-07-07T17:03:10Z

Will this support the commonjs_strict option for the protoc JS generator? The exports when you set the generator to use strict mode are slightly different than when you use commonjs?

Other ts generators also have this problem. I've made an issue on ts-protoc-gen as well related to this:

improbable-eng/ts-protoc-gen#237

murgatroid99 · 2020-07-07T17:05:27Z

This has no relationship to protoc. This is a generator for @grpc/proto-loader, which is a separate library.

DBosley · 2020-07-07T17:18:49Z

Ah! I found this via threads related to ts-protoc-gen about how it doesn't support @grpc/grpc-js so I thought it was to support protoc generated code.

If this ts support is added, @grpc/proto-loader might be the solution my team needs. Thank you!

mjpitz · 2020-07-11T17:32:30Z

I've gone down this route a couple of times. I've been contemplating writing a custom plugin to do some of it. Some of the rough edges I've hit is around the context generation part. When code generating protos for proto-loader, the provided context (i.e. -I) plays a semi-important role. I needed to overlay the typescript generation with some bash scripts to handle the overall context generation.

murgatroid99 · 2020-07-11T18:27:58Z

@mjpitz Can you elaborate on the context issues? Is that something that you think isn't well addressed in this proposal, or would benefit from more detail?

I have a mostly complete implementation of this proposal at grpc/grpc-node#1474, and it seems to be working pretty well for my usage, which has multiple separate include directories.

mjpitz · 2020-07-11T21:35:25Z

@murgatroid99: Of course! So I think the proposal scopes this out by requiring clients to specify the options block. This might be mitigated by how clients end up using the tool to produce libraries. I know when I've done multi-language projects, managing contexts across each language can be tricky to do.

Based on cursory view of this doc, it's not clear to me why we export 3 different types (grpctype, servicehandlers, messages). I would expect this to generate something a little closer to the way other languages generate messages and structures. Last I went down this route, I was trying to base it on some of the manual ones I had written by hand. This was nice for many reasons as it produced similar call signatures to other languages.

I'm sure this proposal works but may end up being toilsome for library owners / consumers. For example, I need to unroll the packages twice (one for the messages, one for the handlers, and even once again for registration). From a consumer's perspective, I would expect this to be somewhat captured by the import rather than duplicated in the package. For example, this feels really dry:

import { messages } from "./path/to/package/proto";
messages.path.to.package.proto.XXXX

vs

import { MyMessage } from "./path/to/package/proto";

Again, I feel like a lot of this could be mitigated if owers wrap this in a client library, but it seems like extra work for something a lot of other languages get for free.

Does this make sense?

mjpitz · 2020-07-11T21:37:59Z

L70-node-proto-loader-type-generator.md

+import { ProtoGrpcType, ServiceHandlers, messages } from 'path/to/proto-file-name_proto'; // File generated by the tool is path/to/proto-file-name_proto.d.ts
+import * as protoLoader from '@grpc/proto-loader';
+
+const packageDefinition = protoLoader.loadSync('other/path/to/proto-file-name.proto', options);


It's quite common for folks to want to load more than one proto file. It would be good to switch this over to an array instead of a simple string to show how multiple types may be loaded.

This API does accept an array of strings as an alternative to a single string. When loading multiple files you can get the type of the output from grpc.loadPackageDefinition by loading all of the corresponding ProtoGrpcType interfaces from the corresponding generating files and combining them with &.

Also, maybe this isn't clear, but the proposed CLI tool can also accept multiple filenames to generate types for.

mjpitz · 2020-07-11T21:39:09Z

L70-node-proto-loader-type-generator.md

+
+```
+
+The tool will be named `proto-loader-gen-types` and will have this usage:


Will this work as a protoc plugin? Or will it stand on it's own.

This is a standalone tool, distributed in the @grpc/proto-loader package. It uses Protobuf.js rather than protoc for parsing.

murgatroid99 · 2020-07-12T05:06:08Z

First, I want to clarify that the runtime exports from the @grpc/proto-loader library already exist, and this proposal would not make any changes to them. This is only about the tool for generating the typescript type definitions to describe the output of those functions. So that API you point to that accepts the options object already exists and won't change.

The 4 exports in the example could probably be consolidated into 3, but fewer would be tricky because of name collisions. To be fair, the messages namespace essentially exports the same information as the individual message files, and the ClientInterfaces and ServiceHandlers namespaces could probably be split out into similar individual files. I think it would probably be cleaner to separate them into separate files.

But the ProtoGrpcType export is different, and needs to live in a root-level file. That's because it is not a namespace aggregating several type definitions, but rather a single type definition that describes the type returned by grpc.loadPackageDefinition.

As for the imports and namespaces, my proposal is a little different from what you describe there. With this proposal, you would be able to write

import { messages } from './generated/protoFilename';
messages.path.to.package.proto.XXXX

but you would actually also be able to write

import { MyMessage } from './generated/path/to/package/proto/MyMessage'

murgatroid99 · 2020-07-13T20:43:37Z

I made some changes to the proposal in response to the recent comments:

The ServiceNameClient and ServiceNameHandler interfaces are now exported from generated/fully/qualified/package/ServiceName.ts, similar to the existing files generated for messages and enums.
The root files now only export the ProtoGrpcType interface. Everything else can be found in other files.

wenbozhu · 2020-07-14T17:31:07Z

L70-node-proto-loader-type-generator.md

+
+The gRPC implementation is specified at build time because the types depend on types from that library and the user should know at that point which implementation they are using. An alternative is to use generics to insert the types in a different way, but that increases the complexity of both the generated code and the code that uses it for relatively little gain.
+
+Two types are generated for each message definition to represent that at runtime the library is permissive with what it accepts, but stricter with what it outputs. Having those stricter output types allows the user to get tigher type guarantees where possible.


This sentence is hard to read. Do you mean the following:

"Two types are generated for each message definition. To represent the message at runtime, the library (tool?) is .... "

What I'm trying to say is that the library is permissive about what types it accepts as input, but we can be more specific about what types it outputs and the two different types reflect that.

wenbozhu · 2020-07-14T17:39:41Z

L70-node-proto-loader-type-generator.md

+ - `--verbose`, `-v`: Enable various logging output
+ - `--includeComments`: Include comments from the `.proto` files in the generated files
+
+The output will be one file for each `message` and `enum` loaded, with file paths based on the package and type names, plus a master file per input file that combines all of those to produce the type that the user will load, as described above. `ProtoGrpcType` types from different files can be intersected to get the type that results from loading those files together at runtime. Messages will have an additional type generated, suffixed with `__Output`, that describes the type of objects that will be output by gRPC, i.e. response messages on the client, and request messages on the server. These "output" message types will be subtypes of the main message type, restricted based on the options that are set.


Could you expand a bit on the need for stricter types, here or under the rationales section?

Also, I wonder if "_Strict" will work better. Conceivably, I may want to use the strict types for both input and output messages, e.g. to improve the overall type conformity of my codebase.

I have reworded the part of the rationales section about the stricter types.

wenbozhu

LG overall. Thanks.

wenbozhu · 2020-07-14T19:29:26Z

L70-node-proto-loader-type-generator.md

+
+The goal of generating two separate interfaces for each message type is to describe two separate things as narrowly as possible: the objects that users can pass to the library as input, and the objects that the library will output. We have more control over what the library outputs, so we can be more specific in that type. This simplifies handling of messages output by the library, while still allowing the same flexibility when providing input messages that users get with the JavaScript interface.
+
+For example, all Protobuf 3 fields are optional, so the input type allows the user to omit fields, but with the `defaults` code generation option, we know that the library will always output the default value for omitted fields, so the output type can guarantee that every field will have a value.


LG. Thanks for the update.

murgatroid99 added 2 commits June 1, 2020 16:46

Add gRFC L70: @grpc/proto-loader TypeScript Type Generator CLI Tool

2852744

Add discussion thread link

69b009d

alexander-fenster reviewed Jun 2, 2020

View reviewed changes

L70-node-proto-loader-type-generator.md Outdated Show resolved Hide resolved

murgatroid99 added 2 commits June 2, 2020 10:25

Fix bug in example code

0b323c6

Add output path argument

8499a61

alexander-fenster reviewed Jun 2, 2020

View reviewed changes

murgatroid99 added 3 commits June 5, 2020 12:54

Describe generated code in detail, add example

785eb9e

Address multiple input files, fix up example types

d81dc49

Make some corrections based on implementing it

791f2e1

murgatroid99 mentioned this pull request Jun 12, 2020

proto-loader: Add TypeScript generator grpc/grpc-node#1474

Merged

murgatroid99 mentioned this pull request Jun 25, 2020

Protobufjs' codegen support grpc/grpc-node#528

Closed

Use actual generated code for examples, add new options

577fa39

mjpitz reviewed Jul 11, 2020

View reviewed changes

Move service definitions into their own file, as suggested

451bb54

wenbozhu reviewed Jul 14, 2020

View reviewed changes

wenbozhu approved these changes Jul 14, 2020

View reviewed changes

Reword reasoning for multiple message interfaces

dedda44

wenbozhu reviewed Jul 14, 2020

View reviewed changes

wenbozhu approved these changes Jul 14, 2020

View reviewed changes

Finalize proposal information

a872c74

murgatroid99 merged commit 195d289 into grpc:master Jul 21, 2020

tatemz mentioned this pull request Nov 1, 2021

Add plugin param to only generate gRPC types improbable-eng/ts-protoc-gen#292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

L70: @grpc/proto-loader TypeScript Type Generator CLI Tool #183

L70: @grpc/proto-loader TypeScript Type Generator CLI Tool #183

murgatroid99 commented Jun 1, 2020 •

edited

Loading

alexander-fenster Jun 2, 2020

alexander-fenster Jun 2, 2020

murgatroid99 Jun 5, 2020

DBosley commented Jul 7, 2020 •

edited

Loading

murgatroid99 commented Jul 7, 2020

DBosley commented Jul 7, 2020 •

edited

Loading

mjpitz commented Jul 11, 2020

murgatroid99 commented Jul 11, 2020

mjpitz commented Jul 11, 2020 •

edited

Loading

mjpitz Jul 11, 2020

murgatroid99 Jul 12, 2020

mjpitz Jul 11, 2020

murgatroid99 Jul 12, 2020

murgatroid99 commented Jul 12, 2020

murgatroid99 commented Jul 13, 2020

wenbozhu Jul 14, 2020

murgatroid99 Jul 14, 2020

wenbozhu Jul 14, 2020

murgatroid99 Jul 14, 2020

wenbozhu left a comment

wenbozhu Jul 14, 2020


		```

		The tool will be named `proto-loader-gen-types` and will have this usage:


		The gRPC implementation is specified at build time because the types depend on types from that library and the user should know at that point which implementation they are using. An alternative is to use generics to insert the types in a different way, but that increases the complexity of both the generated code and the code that uses it for relatively little gain.

		Two types are generated for each message definition to represent that at runtime the library is permissive with what it accepts, but stricter with what it outputs. Having those stricter output types allows the user to get tigher type guarantees where possible.


		The goal of generating two separate interfaces for each message type is to describe two separate things as narrowly as possible: the objects that users can pass to the library as input, and the objects that the library will output. We have more control over what the library outputs, so we can be more specific in that type. This simplifies handling of messages output by the library, while still allowing the same flexibility when providing input messages that users get with the JavaScript interface.

		For example, all Protobuf 3 fields are optional, so the input type allows the user to omit fields, but with the `defaults` code generation option, we know that the library will always output the default value for omitted fields, so the output type can guarantee that every field will have a value.

L70: @grpc/proto-loader TypeScript Type Generator CLI Tool #183

L70: @grpc/proto-loader TypeScript Type Generator CLI Tool #183

Conversation

murgatroid99 commented Jun 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DBosley commented Jul 7, 2020 • edited Loading

murgatroid99 commented Jul 7, 2020

DBosley commented Jul 7, 2020 • edited Loading

mjpitz commented Jul 11, 2020

murgatroid99 commented Jul 11, 2020

mjpitz commented Jul 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

murgatroid99 commented Jul 12, 2020

murgatroid99 commented Jul 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenbozhu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

murgatroid99 commented Jun 1, 2020 •

edited

Loading

DBosley commented Jul 7, 2020 •

edited

Loading

DBosley commented Jul 7, 2020 •

edited

Loading

mjpitz commented Jul 11, 2020 •

edited

Loading