L70: @grpc/proto-loader TypeScript Type Generator CLI Tool

L70-node-proto-loader-type-generator.md

@@ -0,0 +1,218 @@
+@grpc/proto-loader TypeScript Type Generator CLI Tool
+----
+* Author(s): murgatroid99
+* Approver: wenbozhu
+* Status: Final
+* Implemented in: Node.js
+* Last updated: 2020-06-21
+* Discussion at: https://groups.google.com/g/grpc-io/c/FwLprMC-5Qc
+
+## Abstract
+
+Add a tool to the `@grpc/proto-loader` to generate types that describe the objects that will be generated by loading the output of `@grpc/proto-loader` into `grpc` or `@grpc/grpc-js`.
+
+## Background
+
+`@grpc/proto-loader` outputs objects with types determined by the `.proto` files loaded at runtime, so it is currently very difficult to get compile-time type information about those objects. As TypeScript becomes increasingly popular, that compile-time type information becomes increasingly desirable.
+
+
+### Related Proposals: 
+* L23 Standalone Node.js gRPC+Protobuf.js Library API
+* L43 Node Message Type Information
+
+## Proposal
+
+In the `@grpc/proto-loader` library, provide a command line tool that will generate the types of the objects that will be created by passing the output of `load` or `loadSync` to `grpc.loadPackageDefinition` when loading a specific set of `.proto` files with a specific set of options. This will allow users to write code like this to use the type information in the rest of the code:
+
+```ts
+import * as grpc from 'grpc';
+// OR 
+import * as grpc from '@grpc/grpc-js';
+
+import { ProtoGrpcType } from 'generated/proto-file-name_proto'; // File generated by the tool is generated/proto-file-name_proto.ts
+import * as protoLoader from '@grpc/proto-loader';
+
+import { MessageName } from 'generated/fully/qualified/package/MessageName';
+import { ServiceNameHandler } from 'generated/fully/qualified/package/ServiceName';
+
+const packageDefinition = protoLoader.loadSync('other/path/to/proto-file-name.proto', options);
+const loadedPackageDefinition = grpc.loadPackageDefinition(packageDefinition) as unknown as ProtoGrpcType; // How the types will be used
+
+const requestObject: MessageName = {
+  field1: value1,
+  field2: value2
+};
+
+const serviceHandler: ServiceNameHandler = {
+  // Call implicitly has the correct arity (unary, client streaming, etc.) and the call and callback have the correct request and response types
+  methodName(call, callback) {
+    // Implementation
+  }
+}
+
+const server = new grpc.Server();
+server.addService(loadedPackageDefinition.fully.qualified.package.ServiceName.service, serviceHandler);
+
+```
+
+The tool will be named `proto-loader-gen-types` and will have this usage:
+
+```
+proto-loader-gen-types [OPTIONS] <proto file name> ...
+```
+
+The options will correspond directly to the `protoLoader.load` options as follows:
+
+ - `--keepCase`: Preserve the case of field names
+ - `--longs=String|Number`: Specify the type that should be used to output 64 bit integer values
+ - `--enums=String`: Specify that enum values should be output as strings
+ - `--bytes=Array|String`: Specify the type that should be used to output `bytes` fields
+ - `--defaults`: Indicates that default values should be output for missing fields
+ - `--arrays`: Indicates that empty arrays should be output for missing repeated fields even if `--defaults` is unset
+ - `--objects`: Indicates that empty objects should be output for missing message fields even if `--defaults` is unset
+ - `--oneofs`: Indicates that virtual "oneof" fields will be set to the present field's name in the output
+ - `--includeDirs=<directory>`, `-I <directory>`: A directory to search for included `.proto` files. Can be passed multiple times to include multiple directories
+
+ - `--outDir=<directory>`, `-O <directory>`: The directory in which to output files
+ - `--grpcLib=grpc|@grpc/grpc-js`: The gRPC implementation library that these types will be used with
+ - `--verbose`, `-v`: Enable various logging output
+ - `--includeComments`: Include comments from the `.proto` files in the generated files
+
+The output will be one file for each `message` and `enum` loaded, with file paths based on the package and type names, plus a master file per input file that combines all of those to produce the type that the user will load, as described above. `ProtoGrpcType` types from different files can be intersected to get the type that results from loading those files together at runtime. Messages will have an additional type generated, suffixed with `__Output`, that describes the type of objects that will be output by gRPC, i.e. response messages on the client, and request messages on the server. These "output" message types will be subtypes of the main message type, restricted based on the options that are set.
+
+To support this generated code, `@grpc/proto-loader` will need to re-export the `Long` type from `protobufjs`, because that is a type that can be used by generated message types.
+
+### Example Generated Code
+
+With options that will probably be common: `--keep-case`, `--longs=String`, `--enums=String`, `--defaults`, `--oneofs`, and `--grpcLib=@grpc/grpc-js`
+
+#### Input
+
+```protobuf
+// filename.proto
+syntax = "proto3";
+
+package package_name.subpackage_name;
+
+enum EnumName {
+  OPTION0 = 0;
+  OPTION1 = 1;
+}
+
+message MessageName {
+  string string_value = 1;
+  int32 number_value = 2;
+  EnumName enum_value = 3;
+  int64 long_value = 4;
+  oneof oneof_value {
+    bool bool_value = 5;
+    bytes bytes_value = 6;
+  }
+}
+
+service ServiceName {
+  rpc Method (MessageName) returns (MessageName);
+}
+```
+
+#### Output
+
+```ts
+// package_name/subpackage_name/EnumName.ts
+
+export enum EnumName {
+  OPTION0 = 0,
+  OPTION1 = 1,
+}
+
+// package_name/subpackage_name/MessageName.ts
+
+import { EnumName as _package_name_subpackage_name_EnumName } from '../../package_name/subpackage_name/EnumName';
+import { Long } from '@grpc/proto-loader';
+
+export interface MessageName {
+  'string_value'?: (string);
+  'number_value'?: (number);
+  'enum_value'?: (_package_name_subpackage_name_EnumName | keyof typeof _package_name_subpackage_name_EnumName);
+  'long_value'?: (number | string | Long);
+  'bool_value'?: (boolean);
+  'bytes_value'?: (Buffer | Uint8Array | string);
+  'oneof_value'?: "bool_value"|"bytes_value";
+}
+
+export interface MessageName__Output {
+  'string_value': (string);
+  'number_value': (number);
+  'enum_value': (keyof typeof _package_name_subpackage_name_EnumName);
+  'long_value': (string);
+  'bool_value'?: (boolean);
+  'bytes_value'?: (Buffer);
+  'oneof_value': "bool_value"|"bytes_value";
+}
+
+// package_name/subpackage_name/ServiceName.ts
+
+import * as grpc from '@grpc/grpc-js'
+import { MessageName as _package_name_subpackage_name_MessageName, MessageName__Output as _package_name_subpackage_name_MessageName__Output } from '../../package_name/subpackage_name/MessageName';
+
+export interface ServiceNameClient extends grpc.Client {
+  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  Method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, options: grpc.CallOptions, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+  method(argument: _package_name_subpackage_name_MessageName, metadata: grpc.Metadata, callback: (error?: grpc.ServiceError, result?: _package_name_subpackage_name_MessageName__Output) => void): grpc.ClientUnaryCall;
+
+}
+
+export interface ServiceNameHandlers {
+  Method(call: grpc.ServerUnaryCall<_package_name_subpackage_name_MessageName, _package_name_subpackage_name_MessageName__Output>, callback: grpc.sendUnaryData<_package_name_subpackage_name_MessageName__Output>): void;
+
+}
+
+// filename.ts
+
+import * as grpc from '@grpc/grpc-js';
+import { ServiceDefinition, EnumTypeDefinition, MessageTypeDefinition } from '@grpc/proto-loader';
+
+import { ServiceNameClient as _package_name_subpackage_name_ServiceNameClient } from './package_name/subpackage_name/ServiceName';
+
+type ConstructorArguments<Constructor> = Constructor extends new (...args: infer Args) => any ? Args: never;
+type SubtypeConstructor<Constructor, Subtype> = {
+  new(...args: ConstructorArguments<Constructor>): Subtype;
+}
+
+export interface ProtoGrpcType {
+  package_name: {
+    subpackage_name: {
+      EnumName: EnumTypeDefinition
+      MessageName: MessageTypeDefinition
+      ServiceName: SubtypeConstructor<typeof grpc.Client, _package_name_subpackage_name_ServiceNameClient> & { service: ServiceDefinition }
+    }
+  }
+}
+```
+
+## Rationale
+
+The goal is to have type information available when editing and building code that uses `@grpc/proto-loader` to load `.proto` files at runtime. It is not possible to infer this type information from the interfaces currently provided by this library because TypeScript cannot infer type information from `.proto` files. The only other option is to generate this type information separately. The recommended usage of the `@grpc/proto-loader` library is to call `load` or `loadSync` and then pass the result of that to `grpc.loadPackageDefinition`, so the type of the final result of that will be useful to most users. The output of `load` and `loadSync` are objects with runtime data that describes the resulting types, so the types of those objects are significantly less useful, and the final type that we will generate here cannot be effectively inferred from those types.
+
+The type parameters `grpc.Client`, `grpc.Metadata`, and `grpc.CallOptions` are all needed because each of those types is used in the resulting type, and none of them can be inferred from the others. The type `grpc.Metadata` also impacts the resulting type, but its type is the same across the two implementations so it can be defined independently of them.
+
+These alternatives were considered to generating files in a directory structure based on the protobuf package structure:
+
+ - **Generate files in a directory structure mirroring the directory structure of the input files, all relative to the output directory.** This introduces the complexity of handling unusual import paths, including absolute paths and paths with `..`.
+ - **Generate files corresponding to the input files, all directly in the output directory.** This greatly increases the risk of filename conflicts.
+ - **Generate a single file with all of the types.** This creates a large, unwieldy file, making it hard for a human to evaluate changes. This can also greatly increase code duplication if multiple files need to be generated.
+
+The gRPC implementation is specified at build time because the types depend on types from that library and the user should know at that point which implementation they are using. An alternative is to use generics to insert the types in a different way, but that increases the complexity of both the generated code and the code that uses it for relatively little gain.
+
+The goal of generating two separate interfaces for each message type is to describe two separate things as narrowly as possible: the objects that users can pass to the library as input, and the objects that the library will output. We have more control over what the library outputs, so we can be more specific in that type. This simplifies handling of messages output by the library, while still allowing the same flexibility when providing input messages that users get with the JavaScript interface.
+
+For example, all Protobuf 3 fields are optional, so the input type allows the user to omit fields, but with the `defaults` code generation option, we know that the library will always output the default value for omitted fields, so the output type can guarantee that every field will have a value.
+
+## Implementation
+
+I (murgatroid99) will implement this in the `@grpc/proto-loader` library in [PR #1474](https://github.com/grpc/grpc-node/pull/1474)