Why the separate wrap/unwrap from the to/fromJSON methods

[bouk]

Hello @stephenh!

One thing I don't understand about this project is why the wrap/unwrap methods are generated for some Messages like google.protobuf.Value. Why not just return the canonical JSON representation from the to/fromJSON methods and call that recursively when serializing?

[stephenh]

Hi @bouk ! I'm not sure actually... @boukeversteegh wrote most of our wrap/unwrap code ... I could poke around to refresh my memory, but as a lazy ask @boukeversteegh wdyt of @bouk 's question? :-) Thanks!

[boukeversteegh]

the core reason for needing some sort of conversion step, is that some protobuf messages have a json representation that is different from their structure. so we have 4 representations of the same message.

• canonical typescript representation: we have tried as much as possible to make the objects intuitive within typescript. such as optional values having ? or value | undefined as a type.
• protobuf message format: this is the schema defined by protobuf. for example, a list is message ListValue which has a field Values. this is different from how we represent it in typescript, where it would just be an array, like string[].
• json encoding representation: in most cases, this is pretty similar to the protobuf message format. for some well known types, there is a 'canonical representation'. such as the list mentioned before, or the 'Value' type, which in protobuf uses a "switch" to select the field, and there is one field for each type. But in JSON it's just represented as the representative type in json. so instead of sending { stringValue: "foobar"} you send "foobar". note that not all (serverside) libraries do this, so you may want to be able to parse the original 1-to-1 json encoding of the protobuf message, just in case.
• protobuf binary encoding. here there are no smart representations of well known types. the values are encoded as the underlying well known type, following the original field numbers and structure.

by having a separate functions to convert a protobuf message either into typescript, or json, we can represent the object in all these ways.

• wrap/unwrap (not most clear name, admittedly) is to convert TS to the protobuf message schema, this is needed for binary encoding
• toJson/fromJson is needed when handling server generated messages. we need this because unset fields may be omitted, but protobuf requires that all fields are available on a protobuf message, whether they are set or not.

I remember there are more nuances to this that what I explained above. If you really want to get into it, I suggest trying out what you had in mind, and then running the tests, to see where things will break.

[stephenh]

Hi @boukeversteegh ! Thanks for the great reply! That was not as clear in my head as you laid out, so really appreciate that.

So, if I'm following, a tldr is that ts-proto could generate types that match the the JSON encoding 1-for-1, but we've generally chosen "more idiomatic" TS than maybe what the JS spec says.

Actually, phrasing it that way, reminds me of our useJsonWireFormat option, which several users requested b/c it allows taking protobuf-generated JSON straight off the wire and just immediately casting it to the ts-proto-generated type, vs. going through a fromJson translation step.

So perhaps that is what you're looking for?

Either way, @bouk does that answer your question?

[boukeversteegh]

You're welcome!

Let's look at a simple protobuf message to clarify the use of wrapping and json methods.

ts-proto/integration/simple/simple.proto

Lines 16 to 34 in 5ec618c

	message Simple {
	// Name field
	string name = 1;
	/* Age */
	int32 age = 2;
	google.protobuf.Timestamp created_at = 9; // This comment will also attach
	Child child = 3;
	StateEnum state = 4;
	repeated Child grand_children = 5;
	repeated int32 coins = 6;
	repeated string snacks = 7;
	repeated StateEnum old_states = 8;
	// A thing (imported from thing)
	ImportedThing thing = 10;
	repeated bytes blobs = 11;
	google.type.Date birthday = 12;
	bytes blob = 13;
	bool enabled = 14;
	}

JSON conversion

fromJSON does a few things:

• Copy the JSON object's field values to the corresponding Typescript object fields. 👈 Trivial
• Convert JSON encoded values to Typescript native values (such as datetime and enums). 👈 This is the idiomatic part.
• Initialize missing fields to their default value (e.g. "" for string, 0 for int64) 👈 This is required by protobuf specs.

ts-proto/integration/simple/simple.ts

Lines 508 to 527 in 5ec618c

	fromJSON(object: any): Simple {
	return {
	name: isSet(object.name) ? String(object.name) : "",
	age: isSet(object.age) ? Number(object.age) : 0,
	createdAt: isSet(object.createdAt) ? fromJsonTimestamp(object.createdAt) : undefined,
	child: isSet(object.child) ? Child.fromJSON(object.child) : undefined,
	state: isSet(object.state) ? stateEnumFromJSON(object.state) : 0,
	grandChildren: Array.isArray(object?.grandChildren)
	? object.grandChildren.map((e: any) => Child.fromJSON(e))
	: [],
	coins: Array.isArray(object?.coins) ? object.coins.map((e: any) => Number(e)) : [],
	snacks: Array.isArray(object?.snacks) ? object.snacks.map((e: any) => String(e)) : [],
	oldStates: Array.isArray(object?.oldStates) ? object.oldStates.map((e: any) => stateEnumFromJSON(e)) : [],
	thing: isSet(object.thing) ? ImportedThing.fromJSON(object.thing) : undefined,
	blobs: Array.isArray(object?.blobs) ? object.blobs.map((e: any) => bytesFromBase64(e)) : [],
	birthday: isSet(object.birthday) ? DateMessage.fromJSON(object.birthday) : undefined,
	blob: isSet(object.blob) ? bytesFromBase64(object.blob) : new Uint8Array(0),
	enabled: isSet(object.enabled) ? Boolean(object.enabled) : false,
	};
	},

Wrap

The converted simple.proto message does use any wrap methods, because it doesn't use well-known-types like ListValue or Value. value.proto has fields that are well known values, so before encoding, it needs to convert the Typescript representation (which is close to the canonical json format) to the underlying protobuf message structure.

ts-proto/integration/value/value.proto

Lines 6 to 12 in 5ec618c

	message ValueMessage {
	google.protobuf.Value value = 1;
	google.protobuf.ListValue anyList = 2;
	repeated google.protobuf.Value repeatedAny = 3;
	repeated google.protobuf.StringValue repeatedStrings = 4;
	google.protobuf.Struct structValue = 5;
	}

The encode method looks like:

ts-proto/integration/value/value.ts

Lines 21 to 38 in 5ec618c

	encode(message: ValueMessage, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
	if (message.value !== undefined) {
	Value.encode(Value.wrap(message.value), writer.uint32(10).fork()).ldelim();
	}
	if (message.anyList !== undefined) {
	ListValue.encode(ListValue.wrap(message.anyList), writer.uint32(18).fork()).ldelim();
	}
	for (const v of message.repeatedAny) {
	Value.encode(Value.wrap(v!), writer.uint32(26).fork()).ldelim();
	}
	for (const v of message.repeatedStrings) {
	StringValue.encode({ value: v!! }, writer.uint32(34).fork()).ldelim();
	}
	if (message.structValue !== undefined) {
	Struct.encode(Struct.wrap(message.structValue), writer.uint32(42).fork()).ldelim();
	}
	return writer;
	},

You can see that for all the 'special' fields (well known types), an additional 'wrap' step is needed. Here's Value.wrap

ts-proto/integration/value/google/protobuf/struct.ts

Lines 402 to 420 in 5ec618c

	wrap(value: any): Value {
	const result = createBaseValue();
	if (value === null) {
	result.nullValue = NullValue.NULL_VALUE;
	} else if (typeof value === "boolean") {
	result.boolValue = value;
	} else if (typeof value === "number") {
	result.numberValue = value;
	} else if (typeof value === "string") {
	result.stringValue = value;
	} else if (Array.isArray(value)) {
	result.listValue = value;
	} else if (typeof value === "object") {
	result.structValue = value;
	} else if (typeof value !== "undefined") {
	throw new Error("Unsupported any value type: " + typeof value);
	}
	return result;
	},

Here you can see clearly that what is a simple variable in Typescript, will be converted into an object and assigned to the field for the type of that variable.

---

So in short:

• json methods are needed to make sure all implicit fields are initialized, and json values are mapped to typescript values.
• wrap methods are needed to convert the well-known-types to and from their canonical representation