-
Notifications
You must be signed in to change notification settings - Fork 15.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kotlin support request #3742
Comments
Right now it is not planned. We recommend using the Java bindings instead given the interoperability, mostly due to the code size bloat when both kotlin and Java are generated in the same app. Will keep this posted when anything changes. |
Kotlin support would be so much better than the current Java builders. :) |
Don't forget that Kotlin can compile to non-jvm environments, so just saying 'use java' actually doesn't solve the kotlin problem. |
If a Kotlin code generator is ever created, it would be great if "oneofs" were turned into sealed classes, rather than having the "*Case" enum. Also, given how nice null handling is in Kotlin, I wish optional fields could simply be implemented as nullable properties, rather than having the separate "has*" methods. |
Btw kotlinx.serialization supports protobuf already without any code generation. Sent with GitHawk |
@christophsturm that's interesting, but it doesn't seem like it really addresses the issue. The problem is that you usually start with a .proto file that you really do want to use for code generation, and it looks like there's only partial support for protobuf anyway - Kotlin/kotlinx.serialization#67. |
yes it has a different focus and i would love to see kotlin code generation in protobuf. |
I am toying around with implementing this. The ideal situation would be to generate all the required data classes (yes, that means immutable and sealed hierarchies for oneofs) and abstract the sizing, marshalling, and unmarshalling via multiplatform expect/actual classes and defer to existing protobuf serialization code for the three primary targets (JVM, JS, native). Maybe could have hand written serialization, but the existing language impls are highly optimized, especially in the JVM case. Also, for I will update if/when complete. |
Initial version complete: https://github.com/cretz/pb-and-k. I'll broadcast to Kotlin forum and/or Reddit tomorrow so no need to clutter up this issue discussing. It only has basic features now, but I'll implement more if the project becomes popular. |
An another projet looks promising kroto-plus |
This is a project we are (and specifically I am) working on for Google, with an unstable prototype landing in open source sometime in the late Q2-early Q3 range. Some specifics, just to be clear:
|
@lowasser interesting! Can we expect data class generation rather Java builders? |
Neither. Data classes as in
...which is not actually that different, all things considered, from
|
I've posted https://github.com/lowasser/protobuf/blob/master/kotlin-design.md , which is a complete description of the design our internal process came up with. |
@lowasser it’s really great to see this starting to gain some attention. I author a library that provides dsl generation for proto messages https://github.com/marcoferrer/kroto-plus I still think it falls short of true first party kotlin support though. A couple of notes I wanted to add. One thing we experienced early on was the need to have message types annotated with kotlins DslMarker annotation. This provides compile time safety so that one nested lambda doesn’t mutate the scope of a parent lambda. Also I saw that you mentioned that there is a separate discussion for improvements to grpc apis. Is this public yet? The kroto project has been working on refining grpc integration with coroutines and I wouldn’t mind helping contribute |
@lowasser I understand the design constraints might be different for google's kotlin generator (for instance with regards to positional args, data classes, or a copy method), but if your team hasn't yet, it might be helpful to take a look at the approach @thesamet took for ScalaPB:
Although the target language for ScalaPB is different, some of the ideas would translate fairly well to a Kotlin code generator. |
@marcoferrer No, the gRPC design isn't public yet. I am aware of kroto+, and have reviewed its design, and I think we'll be doing something along similar lines, but with a number of small differences -- including keeping the proto enhancements separate from the gRPC enhancements. Please feel free to contact me personally, though, and we can talk about that. I'd prefer to keep the proto issue to the proto discussion :) |
@dsilvasc Point by point:
We already do surface repeated fields as immutable lists and map fields as immutable maps. (I don't believe there's currently a way to expose the unmodifiable interface while keeping the base proto in Java, unless there's an annotation available for that...?) Optional scalars is something we may well do for proto2, though we're mostly focusing on proto3 right now. (If I didn't list that in the "Open Questions" section, that was an oversight on my part.)
|
@lowasser - can you clarify why sealing oneofs inhibits proto evolution? In ScalaPB, we generate an |
@thesamet In Kotlin, my understanding is that an unhandled case in a when-expression is an error, not a warning. When expressions are required to be exhaustive. That makes adding new cases a source-incompatible change. (Demo: https://pl.kotl.in/yWJloqTKg, for a scenario where code that worked before the addition of |
(I'm actually a little surprised to hear it's not an error in Scala -- what actually happens when there's an unhandled case in a match expression?) |
@lowasser Thanks for clarifying. In Scala you get a warning at compile time and a Taking an optional field removal as a potential proto evolution step: this breaks source compatibility for all compiled languages since getters/setters will not be generated. I personally find it desirable to get a compiler warning/error at the use-site when a case or a field is added or removed so I know everything is being handled. Also is scala, it is easy to fix for future unknown cases by pattern matching on |
We've never supported optional field removal as a source-compatible change -- do any language proto APIs? -- but supporting adding new fields without breaking existing code's compilation has always been a business-critical use case for us. As far as I know, all the other officially supported proto APIs have similar support. It's easy to handle unknown future cases in Kotlin, too -- just have an |
Giving users an option to represent oneofs as sealed would be helpful for teams who want their source to break when they update their schemas -- teams who want to be forced to handle new possible cases. The java code generated for enums already does this and it's helpful that teams can choose on a case-by-case basis whether they want source-breaking exhaustive matches or source-forward-compatible |
An option to make oneofs effectively sealed seems like a reasonable addition, though in some ways I wonder if that should really be a Kotlin-exclusive feature. Do other languages for protobufs already have such an annotation? Or was it suggested, and rejected? |
I don't think sealed hierarchies are easily expressible in the languages targeted by this repo (Java, C++, Python, ObjC, C#, JS, Ruby). In languages with sealed types (Scala, Rust, Swift), their protobuf code generators opted for representing oneofs as sealed: https://github.com/apple/swift-protobuf/blob/master/Documentation/API.md#oneof-fields |
Sorry, we don't have time to support this in the near future. Close this for now. |
There is an official initial release, you can check it here https://github.com/grpc/grpc-kotlin. |
This is the part to generate Kotlin gRPC services, right? The generated messages are still Java, or am I wrong? |
There have been some significant hangups; the protobuf team has expressed interest in owning this in the long term and making Kotlin a fully supported language for protos, but requires the code generation to be rewritten in C++ for a number of reasons. That work is ongoing. Additionally, we're continuing to work on both optimization and on future-proof-ness; for example, we're leaning towards dropping the oneof class feature at least for the initial release, because we think it's probable that Java will want to add the same feature down the line and we'd very much want to avoid that breaking compatibility. The related but separate project of Kotlin coroutine APIs for gRPC did indeed release yesterday, but that's not the same thing. |
It looks like Wire recently switched to Kotlin Multiplatform. They don't say anything about JS or Native support on their website but they are publishing a Kotlin/JS artifact: https://search.maven.org/artifact/com.squareup.wire/wire-runtime-js/3.1.0/jar Another option is https://github.com/streem/pbandk (I'm one of the maintainers). pbandk is a Kotlin Multiplatform protobuf library with support for JVM and JS (and Native support is in review). |
Here is my attempt to enhance protobufs with builders: #5341 I just want to remind you about this possibility in case this technique is forgotten. All it's doing it is generating additional extension functions in import com.example.tutorial.*
import com.example.tutorial.AddressBookProtosDSL.PhoneNumber.buildPhoneNumber
import com.example.tutorial.AddressBookProtosDSL.buildAddressBook
import com.example.tutorial.AddressBookProtosDSL.buildPerson
fun main(args: Array<String>) {
println(buildPerson {
id = 2
name = "fff"
})
println(buildPhoneNumber {
number = "123"
})
println(buildAddressBook {
people {
addPerson {
id = 1234
name = "John Doe"
email = "jdoe@example.com"
dedicatedPhone {
type = AddressBookProtos.Person.PhoneType.WORK
name = "abc"
number = "2222-3333"
}
phones {
addPhoneNumber {
type = AddressBookProtos.Person.PhoneType.HOME
name = "def"
number = "555-4321"
}
addPhoneNumber {
type = AddressBookProtos.Person.PhoneType.WORK
name = "ghi"
number = "555-3421"
}
}
}
}
})
} // Generated by the protocol buffer compiler. DO NOT EDIT!
// source: tutorial.proto
package com.example.tutorial;
object AddressBookProtosDSL {
fun buildPerson(
block: com.example.tutorial.AddressBookProtos.Person.Builder.() -> Unit
) = com.example.tutorial.AddressBookProtos.Person.newBuilder()
.apply(block)
.build()
object PhoneNumber {
fun buildPhoneNumber(
block: com.example.tutorial.AddressBookProtos.Person.PhoneNumber.Builder.() -> Unit
) = com.example.tutorial.AddressBookProtos.Person.PhoneNumber.newBuilder()
.apply(block)
.build()
}
fun buildAddressBook(
block: com.example.tutorial.AddressBookProtos.AddressBook.Builder.() -> Unit
) = com.example.tutorial.AddressBookProtos.AddressBook.newBuilder()
.apply(block)
.build()
// @@protoc_insertion_point(outer_dsl_object_scope)
}
fun com.example.tutorial.AddressBookProtos.Person.Builder.phones(
block: MutableList<com.example.tutorial.AddressBookProtos.Person.PhoneNumber>.() -> Unit
) {
addAllPhones(
mutableListOf<com.example.tutorial.AddressBookProtos.Person.PhoneNumber>()
.apply(block)
)
}
fun com.example.tutorial.AddressBookProtos.Person.Builder.dedicatedPhone(
block: com.example.tutorial.AddressBookProtos.Person.PhoneNumber.Builder.() -> Unit
) {
dedicatedPhoneBuilder.apply(block)
}
fun com.example.tutorial.AddressBookProtos.AddressBook.Builder.people(
block: MutableList<com.example.tutorial.AddressBookProtos.Person>.() -> Unit
) {
addAllPeople(
mutableListOf<com.example.tutorial.AddressBookProtos.Person>()
.apply(block)
)
}
fun MutableList<com.example.tutorial.AddressBookProtos.Person>.addPerson(
block: com.example.tutorial.AddressBookProtos.Person.Builder.() -> Unit
) {
add(
com.example.tutorial.AddressBookProtos.Person.newBuilder()
.apply(block)
.build()
)
}
fun MutableList<com.example.tutorial.AddressBookProtos.Person.PhoneNumber>.addPhoneNumber(
block: com.example.tutorial.AddressBookProtos.Person.PhoneNumber.Builder.() -> Unit
) {
add(
com.example.tutorial.AddressBookProtos.Person.PhoneNumber.newBuilder()
.apply(block)
.build()
)
}
fun MutableList<com.example.tutorial.AddressBookProtos.AddressBook>.addAddressBook(
block: com.example.tutorial.AddressBookProtos.AddressBook.Builder.() -> Unit
) {
add(
com.example.tutorial.AddressBookProtos.AddressBook.newBuilder()
.apply(block)
.build()
)
} |
For those who is still in need for Kotlin protobuf support, we successfully applied pbandk after latest release 0.8.1 which updates the runtime to latest Kotlin and Kotlin serialization. We're able to use same schema library on frontend and backend making it easier to write crossplatform Kotlin code. |
Are there examples of people using pbandk with gRPC servers? There don't seem to be a ton of examples of gRPC servers using the pbandk bindings. |
@pratik-brex There isn't explicit support for gRPC in pbandk. But there is a generic service code generation mechanism which is intended for use cases like gRPC: https://github.com/streem/pbandk#service-code-generation. I'm not aware of anyone actively using it yet, so do let us know your feedback/suggestions if you try it. |
@lowasser It's been over 7 month since your last update on Protobuf team plans for first-class Kotlin support. |
I would also want to see an update here. I'd like to use gRPC in my Kotlin application, yet the not at all idiomatic code generated by protoc-java is really deterring me from using it. Google has made great efforts in embracing Kotlin, especially in the Android world, but just saying that Kotlin is the preferred language for this and that is not enough. Kotlin stands out because of its unique and ergonomic language features. Simply translating Java code into Kotlin completely defies the point of using Kotlin instead of Java. In my opinion, writing and generating Kotlin-idiomatic code instead of Java-like code (as explained in Koans) should be an integral part of promoting Kotlin as, for example, the preferred Android language. This also includes proper Kotlin support in both ProtoBuf and gRPC. Probably one of the most important features that a hypothetical protoc-kotlin compiler should have is to implement the officially promoted type-safe builder pattern instead of a Java-style builder pattern. This has already been addressed in this discussion. Concerning the discussion about sealed classes, a few points seem to be relevant to add:
Anyway, I hope that this feature is still being worked on, because I really think it is a necessity as of today. Since the discussion is already 5 months old, I am not sure about the state of third-party Proto/gRPC for Kotlin projects. If anyone can point me towards the currently active projects, that would be very nice. I think we deserve some kind of update on this feature by now. |
The state is that yes, it is still being worked on extremely actively; no, there is not much to show for it publicly yet. Some things that have happened in the past few months:
|
If you're looking for third-party support, as mentioned above (disclaimer: I'm a developer), https://github.com/open-toast/protokt generates idiomatic Kotlin code and supports ServiceDescriptor and MethodDescriptor generation that is compatible with both grpc-java and grpc-kotlin. It does not interoperate at the Java/Kotlin level with code generated with protobuf-java, which is something the protobuf team is doing. It has the potential for cross-platform functionality but that's not currently in the roadmap. It is used in production at Toast. If you're looking for a solution that does have interop with protobuf-java's generated code, you might look at https://github.com/marcoferrer/kroto-plus. My guess is that it will be mostly straightforward to migrate from that to an official library when that time comes. Note also that you can use grpc-kotlin with the Java bindings that exist today and get coroutine support. Later, when an official Kotlin generator is released, you can migrate message building separately. |
To add another option, https://github.com/streem/pbandk generates idiomatic Kotlin code and supports Kotlin Mulitplatform (JVM, JS, and Native). Disclaimer: I'm the primary maintainer of the project. We use it in production at Streem in our backend and Android code. pbandk is pretty similar in spirit to protokt. I'd say the main differences with protokt are that protokt includes built-in support for gRPC (which pbandk lacks), but pbandk includes support for JSON and Kotlin Multiplatform (which protokt lacks). pbandk also focuses on implementing serialization/deserialization in the runtime library, whereas protokt implements it in the generated code. There are of course lots of smaller differences too. pbandk currently generates Kotlin data classes for each proto message, which are idiomatic Kotlin but run into some backwards-compatibility concerns when the proto message definition evolves (as discussed above). We plan to replace those with Kotlin-style type-safe builders in an upcoming version. |
@lowasser Is there any date that is being aimed for? I think many people would like to support the project, but as long as it's not out in the open, no one can. |
It seems like something is happening in the repo! 17 days ago an interesting PR landed into master! Does this mean that we're that close? 😍 |
Does it support unsigned integers that are now stable in Kotlin 1.5? |
Looking at what landed in master it seems that no Kotlin code is generated, it's all Java and the existing generated code is extended via extension functions and additional wrapper types. I actually feared that we are not going to see idiomatic Kotlin code and only get overhead. If this sounds negative then it does because it is. I would wish so much to have some clean Kotlin code with all its possible beauty over Java, e.g. no more |
See also my recent request to actively support kotlinx.serialization in addition to protobuf in the grpc-kotlin project. Grpc is not strictly tied to protobuf. grpc/grpc-kotlin#255 |
We were holding off on posting specifically about this because we were planning to mention it at I/O, but as that talk has gone up, yes, we’ve launched Kotlin support in protobuf. There were definitely some design decisions that disappointed people – including me – but we think we made them for good reasons. Most of them boil down to the fact that we see Java compatibility as a business requirement: almost all of our Kotlin codebases at Google, and most Kotlin codebases in the world, are mixed with Java. For example, this is why we haven’t adopted unsigned types, and are continuing to use Java hazzers and the like. For many features where Kotlin adds potential value, we wanted to be sure that we have a plan for ensuring compatibility with Java even if Java protos add the same features down the road. For example, Java is finalizing sealed types in September with Java 17, and optionality was added “back” into proto3 as recently as February. We thought it was best to hold off on adding special oneof support via sealed types, or orNull extension properties for optional fields, until these issues were settled. We don't need to wait until those features are actually added to Java protos, but we needed enough information to make a compatibility plan. We've carefully considered a lot of design space, and we will continue to evolve protobufs for Kotlin, likely including the features discussed above. It’s even conceivable that someday we might compatibly convert protobufs for Java to depend on Kotlin, instead of the other way around, to try to achieve an ideal experience for developers of both languages. However, such an effort would have to capture the years of engineering effort that have gone into optimizing protos for Java, especially the “lite runtime” for Android. Those optimizations are mission critical for our applications, so we can’t “just” do a complete rewrite. In any event, we hope you find value in the Kotlin support we’ve released, and we look forward to improving it in future. If you have new feature requests for Kotlin protos, please file those as separate issues. |
Awesome, thank you so much |
please support convert to kotlin
The text was updated successfully, but these errors were encountered: