Serializing a Message to Any

[KaiserKarel]

I'm unable to find any docs on how to create an Any from a Message. Do I need to serialize the message myself and set the typeURL?

[fridge-dev]

+1, I need to deserialize an Any to a known type. In other languages, I see there are pack/unpack methods. I couldn't find such methods on the Any type (https://docs.rs/prost-types/0.6.1/src/prost_types/protobuf.rs.html#964-998 )

[fdeantoni]

Similar to #277.

[cetanu]

Subscribed... I know how to go from a type to Any, but not the other way around.

[petergarnaes]

The tools for de-serializing an Any type is already there, you can match on the type URL, and decode the value field using Message::decode.

With a little bit of macro magic you don't have to write a lot of code, I have made an example repo. IMO this is simpler and can easily be adapted to your own needs.

[vbkaisetsu]

If there is a mechanism for Message to return type_url, it can support generics.

use prost::Message;
use prost_types::Any;

fn to_any<T>(message: &T) -> Any where T: Message {
    Any {
        type_url: T::type_url().to_string(),
        value: message.encode_to_vec(),
    }
}

fn from_any<T>(message: &Any) -> Result<T, DecodeError> where T: Message + Default {
    if &message.type_url == T::type_url() {
        T::decode(message.value.as_slice())
    } else {
        Err(DecodeError::new("Invalid type_url"))
    }
}

[colin-grapl]

Adding a type_url to Message would definitely help. After that it would be straightforward to add pack/unpack methods for messages.

[ghost]

Create a datafusion-proto crate for datafusion protobuf serialization apache/datafusion#1887

@vbkaisetsu is there a type_url for prost::Message? I don't think there is one if not this will be straightforward

[vbkaisetsu]

@nyanchor I previously created #535 to support seamless seriarization, but it needs further discussion.

[LucioFranco]

Yeah, I think supporting this is a good idea. Will need to dig in a bit more to understand how other languages support this feature.

[tarcieri]

@LucioFranco the simplest way to support it is adding an associated const TYPE_URL: &'static str to either the Message trait (or another specialized trait if adding it to Message is too onerous). Or if there are genuinely good use cases, possibly a static method.

Then prost-build just needs to compute and generate it for each type.

[LucioFranco]

@tarcieri yes, though I don't remember (or couldn't find but didn't look super hard) if there was actually a concrete way to generate that url that isn't google specific etc.

[tarcieri]

To my knowledge the type URLs take the form:

/<package>.<type name>

e.g.

/foo.bar.baz.MyTypeName

[kurojishi]

Bumping this one up as it's definitely something that will make my life easier.

[Percivalll]

Do we have any latest process?

[tarcieri]

One way to get started would be to add a trait with a carrier for the type URL:

pub trait TypeUrl: Message {
    const TYPE_URL: &'static str;
}

This would avoid the need to immediately populate it for every Message, but provide a common way to implement methods (e.g. on prost_types::Any) for decoding/encoding Message + TypeUrl types from/to Any.

At least initially, the TypeUrl annotations could be added manually, until such a time that prost-build can emit them automatically.

Edit: opened a PR with this idea: #858

[tarcieri]

I opened #896 as my latest attempt to implement this feature

[journaux]

@tarcieri would you mind indicating why Name + Pkg needed in your implementation ? also, would you mind indicating why type_url() isn't exported by default on Prost::Message ?

I can implement prost-build; however, a better approach would remove those unnecessary traits

fn macros(fds: FileDescriptorSet) {
  for fd in &fds.file {
   .....
    for buf in &fd.message_type {
      ....
      let tokens = quote! {
          impl prost::Name for #bufname {
              const NAME: &'static str = #bufurl;
              const PACKAGE: &'static str = #pkg;
              fn type_url() -> String {
                  #bufurl.to_string()
              }
          }
      };
    }
  }
}

[tarcieri]

would you mind indicating why Name + Pkg needed in your implementation

Please see the discussion here: #858 (comment)

also, would you mind indicating why type_url() isn't exported by default on Prost::Message

There is currently no prost-build support, so trying to do that would break all code generation in addition to all currently generated code, as it would be a breaking change to Message.

In the future when there is prost-build support, it might make sense to combine Name into Message.

[journaux]

so it’s a problem with the design + impl not any technical decision / desire ?

If that’s the case — prost::Message::type_url can trivially be added to prost-build

[tarcieri]

It’s something that could potentially be merged into Message in the next breaking release provided prost-build support were implemented, yes

[andrewhickman]

Merging Name into Message might be problematic for custom Message implementations like prost_reflect::DynamicMessage, unless the string types were changed to something like Cow<'static, str> to allow an owned string.

[tarcieri]

Yeah, the Name trait in its current form is poorly suited to anything dynamic. We could potentially refactor it to make it easier to use for that use case (and potentially trait object safe) in the next release