v1.15.0-rc2

This is still a release candidate due to the significance of changes to under-the-hood machinery. But we expect to cut a proper v1.15.0 soon.

In addition to all of the many changes in v1.15.0-rc1, this release includes the following:

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• Fuzz testing identified a couple of potentially severe performance issues in the underlying protocompile package implementation. These were innocuous-looking issues that could become extreme if given the right kind of pathological input. These issues were fixed, and the fixed version of protocompile has been pulled into this repo.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• If printing descriptors that had no source code info and no other sort configuration was used on the Printer, then elements with multiple option declarations would have those options printed in seemingly random order. The actual order was based on a map iteration, and was thus non-deterministic. This has been fixed. In such a case, options will now be ordered by name.

v1.15.0-rc1

This is a very big change under-the-hood, which is why this is a release candidate. Unfortunately, go get -u ... and tools like Depend-a-bot will not take release candidates/pre-release versions. So this candidate is a sort of shot in the dark, hoping that some brave souls might try it out and provide feedback before I stamp it as an official release.

All of CI is green, I've run apidiff to make sure there are no backwards-incompatible changes in the exported API, and I've done some other exploratory testing to make sure this is all good. My hope is that no one will notice a thing. If they do notice something, it might be a slight performance degradation, but hopefully not consequential enough to be a problem.

I expect v1.15 to be the last v1 version. Any other changes significant enough to warrant a new minor version will instead be added in an upcoming v2 version. This version is intended to be a stop-gap between v1 and v2 as it provides a long-needed bridge between the functionality in this repo and the newer google.golang.org/protobuf/reflect/protoreflect package (and its accompanying packages, protodesc, protoregistry, and dynamicpb).

"github.com/jhump/protoreflect/desc"

Changes/fixes:

• This package, the core of this whole repo, with its descriptor interfaces and related functions, has been substantially overhauled. The descriptor values provided by this package are now backed by the descriptor implementations in the google.golang.org/protobuf/reflect/protoreflect package.
  • The primary advantage of this change is that it is now easy to convert a protoreflect.Descriptor to a desc.Descriptor (via the various Wrap* functions in this package). It is similarly easy to unwrap a desc.Descriptor value, to recover the underlying protoreflectDescriptor.
  • This allows this package to be easily used in conjunction with protoreflect and accompanying packages. You can take a protoreflect.Descriptor and then easily use it with the desc/builder and desc/protoprint packages in this repo. Similarly, you can create a desc.Descriptor using the desc/protoparse package and easily turn that into a protoreflect.Descriptor, for use with the google.golang.org/protobuf/... packages.

Additions:

• This adds a new DescriptorWrapper interface, which is implemented by all descriptor implementations in this package. It contains an Unwrap() protoreflect.Descriptor function, to recover the underlying protoreflect.Descriptor.
• Also, all descriptor implementations in this package also have additional methods that are more strongly typed. For example, *desc.FileDescriptor has a method UnwrapFile() protoreflect.FileDescriptor.
• Finally, this package now has numerous Wrap* functions, which accept a protoreflect.Descriptor and wrap it, returning a desc.Descriptor. There is one function for each concrete type, for example for messages there is WrapMessage(d protoreflect.MessageDescriptor) (*MessageDescriptor, error)

"github.com/jhump/protoreflect/desc/builder"

Changes/fixes:

• Previously, not all rules of the Protobuf language were enforced when building a new descriptor with this package. Since the desc.Descriptor values returned by this package are now backed by protoreflect.Descriptor values, more rules are enforced. This is because the implementation of descriptors in the protoreflect package does perform all of those validation checks. The new checks that are now enforced that previously were not:
  1. Files with a syntax of proto3 are not allowed to have required fields.
  2. Files with a syntax of proto3 are not allowed to have messages that define extension ranges.
  3. Files with a syntax of proto3 are not allowed to use groups.
  4. Files with a syntax of proto3 are not allowed to declare default values for fields.
  5. Extension fields must use tag numbers that are in an extension range defined on the extended message.
  6. Non-extension fields are not allowed to use tags that lie in a message's extension ranges or reserved ranges.
  7. Non-extension fields are not allowed to use names that the message has marked as reserved.
  8. Extension ranges and reserved ranges must not overlap.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• This package has been overhauled perhaps even more than the desc package. The implementation in this package has been completely replaced with the functionality of the github.com/bufbuild/protocompile package. So the exported APIs in this package are now just adapters. The actual parser/compiler is implemented in this other dependency. Most of this adaptation logic is trivial with the exception being for the Parser.ParseToAST method, which must convert from protocompile's AST model to the model defined in desc/protoparse/ast.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• Formatting of message literals in custom option values has been improved.
  • It now encloses nested messages in curly braces ({ and }) instead of angle brackets (< and >), which is better aligned with the commonly-used and preferred syntax style.
  • Also, nested messages now respect the Printer.MessageLiteralExpansionThresholdLength, instead of this threshold only being applied for a top-level message. So small nested messages can be emitted in compact form, even if enclosed within a message literal that has been expanded.

"github.com/jhump/protoreflect/desc/sourceinfo"

Additions:

• A new TypeResolver interface has been added, as well as a GlobalTypes package variable that implements that interface. This is analogous to protoregistry.GlobalTypes, just as the existing sourceinfo.GlobalFiles corresponds to protoregistry.GlobalFiles.
• This adds new Wrap* functions, that accept various protoreflect types and return values that are identical to the input values except that they include source code info that was registered with this package.

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• Previously, if protoreflect APIs were used to store a value in an extension field, using a protoreflect.FieldDescriptor that was not a generated extension (i.e. not known at compile-time), trying to convert the resulting message to a *dynamic.Message would overlook that field; it would be absent from the result, not even appearing in the unrecognized fields. This has been fixed. Such extension values will be in the resulting *dynamic.Message and appear as recognized/known fields.

v1.14.1

"github.com/jhump/protoreflect/desc/builder"

Changes/fixes:

• When building a descriptor, leading detached comments were incorrectly ignored. Now they are correctly included in the resulting descriptors' source code info.

v1.14.0

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• If converting or merging a generated message into a *dynamic.Message, if the generated message contained dynamic extensions (extension fields not known to the Protobuf runtime's global registry), they would be lost during the conversion. This has been fixed.

"github.com/jhump/protoreflect/grpcreflect"

Additions:

• Adds a new grpcreflect.NewClientAuto method, which returns a client that will automatically use either the v1 version of server reflection or v1alpha, depending on what the server supports.
• Adds a new grpcreflect.NewClientAlpha method which is meant to replace grpcreflect.NewClient. It always uses the v1alpha version of server reflection (and its predecessor, grpcreflect.NewClient, is now deprecated since it is unclearly named).

v1.13.0

This version includes mostly bug fixes. It does include some new API and features in desc/builder and desc/sourceinfo packages. This version also introduces support for Windows. (Some packages did not work correctly on platforms where the default path separator was not /. These packages have been fixed.)

"github.com/jhump/protoreflect/desc"

Changes/fixes:

• If using the CreateFileDescriptor functions to convert descriptor protos into "rich" descriptors, a panic could occur on certain kinds of invalid input. For example, if the input type of an RPC method referred to an enum instead of a message, a type assertion panic would be triggered. This has been fixed so that such incorrect input results in the function returning an error, not panic'ing.

"github.com/jhump/protoreflect/desc/builder"

Additions:

• Adds a new method PruneUnusedDependencies to the FileBuilder type. This can be used to remove any unused dependencies. This is mainly useful when modifying an existing descriptor and removing elements from it. After removing elements, some of the file's imports may no longer be used.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• Fixes a potential panic when parsing and linking source files. If the source contains a symbol that conflicted with the name of a synthetic oneof (an implementation detail of how descriptors with "proto3 optional" fields are generated), it would trigger a nil de-reference panic while trying to construct the error message. This has been fixed.
• Fixes a subtle issue when resolving relative references in field types and extendees, whereby this package would previously accept a reference that protoc would reject. The resolution rules in protoc handle partially-qualified references a little differently than unqualified references, and this package didn't previously implement the subtle difference. This is now fixed.
• Fixes an issue where a field type was allowed to directly refer to the name of a synthetic map entry message. This is not actually allowed by protoc. Now such an incorrect usage results in an appropriate error.
• Adds checks to verify that default JSON names for fields and enum values do not conflict. This implements the check in the same fashion as protoc, which is case-insensitive. For enum values, any prefix on the value name that matches the enum name (regardless of case or underscores) is disregarded when performing this check. The check results in an error for proto3 syntax files, but only a warning for proto2 syntax files.
• Adds a check to make sure that custom json_name options on fields are valid. So if two fields try to declare the same json_name, this will be considered an error. For proto3 syntax files, if a custom json_name option conflicts with another field's default JSON name, this is also an error (but only a warning for proto2 syntax files).
• The ParseFilesButDoNotLink method will now return uninterpreted options in a form that matches the way the official C++ Parser class does. In particular, aggregate values now use the same string format, which will be identical to the original message literal in source but without the enclosing braces ({ and }), with all comments and whitespace removed, and with a single space inserted between lexical elements.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• Previously, characters outside US ASCII (7 bit) in string literals would be encoded using octal escapes when printed. (There is not enough information in a descriptor to preserve the original source's encoding.) This made string literals with other unicode code points (other languages/alphabets, emoji, etc) impossible to read. This has been fixed. String literals used for bytes fields still use aggressive octal escapes. But for string fields, escapes are only used for non-printable code points.

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• The GetKnownType method of KnownTypeRegistry could previously return an invalid type, if given the fully-qualified name of a synthetic map entry message. It now returns nil in such a case.

"github.com/jhump/protoreflect/desc/sourceinfo"

Additions:

• This release adds a new RegisterEncodedSourceInfo function, used by newer code generated by protoc-gen-gosrcinfo. It accepts compressed encoded data in the form of a byte array.

"github.com/jhump/protoreflect/desc/sourceinfo/cmd/protoc-gen-gosrcinfo"

Changes/fixes:

• The generated code now emits compressed information. That way, it uses less space in the executable, resulting in smaller binaries.

v1.12.0

This change includes a new feature related to source code information for compiled-in proto descriptors. It also includes some other bug fixes.

"github.com/jhump/protoreflect/desc"

Changes/fixes:

• The FindSymbol method of FileDescriptor would panic if given an empty string as the symbol name. This has been fixed. This case now returns a nil descriptor, indicating no element with the given name could be found.
• The various Load* functions will return descriptors that include comment information (and the underlying FileDescriptorProto will contain source code info) if the protoc-gen-gosrcinfo plugin was used to generate additional Go code for these proto files.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• If an option included a message type, and was printed as a compact message literal, it had an errant extra space before the final closing '}' symbol. This is fixed, and these are now printed with a single space between the last field value and the closing brace.

"github.com/jhump/protoreflect/desc/sourceinfo"

This is a brand new package which allows for registering source code information for a proto source file. It also includes some API for using/querying the source code info. This is useful to retain comments for compiled and linked-in descriptors. It is not expected that users would explicitly register source code info; instead use the protoc-gen-gosrcinfo plugin, in addition to protoc-gen-go, when compiling your proto sources.

"github.com/jhump/protoreflect/desc/sourceinfo/cmd/protoc-gen-gosrcinfo"

This is a brand new command which is a protoc plugin for augmenting compiled and linked-in descriptors with source code information. When used, the desc package can include source code info (and thus comments) when loading linked-in descriptors. The desc/sourceinfo package can be used to query for the source code info and can be useful for integrating this data with other things, such as gRPC server reflection.

v1.11.0

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• If an option on a method included comments and was the kind of option that protoprint is able to handle, they would fail to be included in the printed output. Options were supported on all other types, just not on methods. This has been remedied: methods now have the same support for options comments as other elements.
• The Printer type includes three new fields, to control formatting of complex options: ShortOptionsExpansionThresholdCount and ShortOptionsExpansionThresholdLength control when "short options" will be expanded to multiple lines, based on the number of options and the length of the options when rendered; MessageLiteralExpansionThresholdLength controls when message literals in option values will be expanded to multi-line form, based on if the length of the rendered string is too long.

"github.com/jhump/protoreflect/grpcreflect"

Changes/fixes:

• The LoadServiceDescriptors function now accepts an interface, not just the concrete type *grpc.Server. Since *grpc.Server implements the interface, this change should be mostly backwards compatible. However, if there are usages that rely on the precise signature, such as assigning to a function variable whose signature requires *grpc.Server, this is a minor breaking change. Such usage is not expected.

v1.10.3

This release contains several fixes to the desc/protoparse package and one fix to the grpcreflect package.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• If a custom option value contains a message literal, protoc accepts identifiers t, f, True, and False as synonyms for true and false. Use of these alternate spellings would be rejected by this package. This is now fixed so that this package accepts the same values as protoc.
• If a custom option refers to an enum which has values named true or false, this package would reject the option, misunderstanding the true or false identifier to be a boolean literal instead of an enum value name. This has been fixed.
• There were several cases where references to a custom option or extension, in an option name or in a message literal, were resolved differently by this package than by protoc. This lead to some variances -- some source files that protoc would accept but that this package would not (because it was unable to resolve a reference), and some source files that this package would accept but that protoc would reject (because this package was using different lexical scoping rules during resolution). This has been fixed and this package now resolves extension names the same way as protoc.
• When specifying a custom option value for a message whose type is google.protobuf.Any, protoc supports a special syntax that makes it possible to use a simple text format for the contained message, instead of having to include a byte string representation of a marshaled/encoded value. This package did not previously implement that syntax, so would reject proto sources that used it. This has been remedied, and the special syntax is now supported by this package.
• This package would previously accept a repeated extension for a message that used message-set wire format. However, only optional extensions are allowed for such messages. This has been fixed, and proto sources that try to define such a repeated extension will be rejected.
• Extensions are not allowed to set a json_name option, however this package was accepting proto sources that did so. This has been fixed, and proto sources that define an extension with the json_name option will be rejected.

"github.com/jhump/protoreflect/grpcreflect"

Changes/fixes:

• In some cases, servers implementing the reflection service has been observed to incorrectly include extra file descriptors in response to a file_containing_symbol request. Also, the reflection service does not actually specify any ordering requirements for responses that choose to include more than one file. But this package mistakenly assumed an ordering (based on an older implementation of the reflection service in the official Java runtime), which could cause such cases (responses with multiple or even superfluous files) to return the incorrect file descriptor. This has been fixed. Now all responses to file_containing_symbol, file_containing_extension and file_by_filename requests correctly support multiple files (even superfluous ones) in any order.

v1.10.2

This release contains several fixes to the desc/protoparse and dynamic packages.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• If a custom option has a type with a oneof in it, then option values in proto sources should not be allowed to set more than one of the fields. Previously, this package would accept such sources, and only the last field (in source location order) would end up being set in the option value. Now the package will report an error.
• This package would incorrectly accept numeric literals in formats that the Go language accepts but that the Protocol Buffer does not. For example: "0b0100", "0o1735", and "1_000_000" are not legal integer literals in proto source, but are legal Go integer literals and were incorrectly accepted by this package. Similarly, Go accepts "1_000.314" as a float literal and so did this package. But this is not allowed in proto source. These issues have been fixed and this package now accepts numeric literals using the same rules as protoc.
• This package would report an error if a numeric literal over-flowed what can be represented by a 64-bit floating point value (IEEE754 double precision). However, protoc accepts such sources and, like other floating point literals that cannot be perfectly represented, picks the nearest/best option. In the case of overflow, that value is infinity. This package now behaves like protoc and substitutes an infinite value for numeric literals that would otherwise overflow a 64-bit floating point value.
• A NUL character in a source file (code point zero, 0x00) could be misinterpreted as end-of-file. This character is not allowed in proto sources, but would not only be accepted in some cases but also result in incomplete compilation. Similarly, this package accepted such characters inside of comments, but that is also not allowed. This has been remedied and source files with such characters will be rejected.
• This package would incorrectly allow an integer literal like "09", treating it as decimal. However, in proto sources, an integer literal that starts with a zero must be octal (in the given example, "9" is out of range and should have resulted in error). This is fixed.
• This package would allow reserved ranges and extension ranges to start at zero. However one is lowest acceptable value for such ranges. This has been fixed.
• This package would incorrectly allow two different oneofs in the same message to have the same name. This has been fixed.
• This package would allow proto source files that try to directly import the same file more than once. While it is allowed for the transitive dependency graph of a file to include a file more than once, a single proto source file cannot reference the same file multiple times. This is fixed and now reported as an error.
• Custom option values that used the message literal syntax were required to use a colon separator between a repeated field name and an array literal (e.g. multiple values inside of [ and ]). However, protoc does not require the colon in this context. This package has been updated to match the behavior of protoc.
• Parsing a source file could lead to a runtime panic under some conditions of symbol collisions. For example, if an operation used a built-in/well-known import (such as google/protobuf/descriptor.proto) without supplying its own copy of source and then included a source file that defined a symbol with the same fully-qualified name as an element in that import, a panic would occur. This has been fixed and the conflict is reported as an error as expected.
• This package previously populated trailing comments differently from protoc for block elements (i.e. those with elements defined inside of brances { }, including messages, groups, oneofs, extend blocks, enums, services, and methods). This package used the comment immediately following the closing brace (}) as the element's trailing comment. However, protoc uses the comment immediately following the opening brace ({) as the trailing comment. The behavior in this package has been changed to match that of protoc.

"github.com/jhump/protoreflect/desc/protoprint"

Changes/fixes:

• This package previously assumed that trailing comments for block elements (i.e. those with elements defined inside of brances { }, including messages, groups, oneofs, extend blocks, enums, services, and methods) should be rendered after the closing brace. However, this is not how trailing comments are parsed by protoc: they are the comment immediately following the opening brace instead. So this package has been updated to render the trailing comment for these elements after the opening brace.

"github.com/jhump/protoreflect/dynamic"

Changes/fixes:

• When unmarshaling JSON into a dynamic message, where one of the fields is of type google.protobuf.Value, this package would incorrectly process values that were JSON arrays. This should be allowed and result in the field's list_value field being set with the contents of the array. Instead, the field would be set to a single value equal to the last value in the JSON array. This has been fixed.
• When marshaling a dynamic message to JSON, if a field used a custom JSON name that included a character that was non-printable or outside of the 7-bit range (e.g. a control character, code points 0-31, or any code point >= 127), the result JSON output would be invalid. The characters would be encoded in a way such that the resulting key was not a valid JSON string. This has been fixed.

v1.10.1

This release contains some fixes to the protoparse package.

"github.com/jhump/protoreflect/desc/protoparse"

Changes/fixes:

• Use of "groups" (an old feature only supported in syntax proto2) in custom options did not work correctly. The parser would incorrectly reject use of groups in some cases. And when using message literals as option values, it did not accept the same input as protoc with regards to the field name used to refer to a group value. These issues have been corrected and behavior is not consistent with protoc.