Questions on language binding spec

[translatenix]

1. When decoding a message, is the order of properties guaranteed?
2. When a new property is added to a message, is it guaranteed to be added at the end?
3. When decoding a Typed, is the order of properties guaranteed?
4. When decoding a Dynamic, is the order of elements guaranteed?

Every "yes" will make decoding easier and more efficient.

[bioball]

When decoding a message, is the order of properties guaranteed?

The order of properties in the MessagePack message body does not have an order guarantee. A compliant decoder should decode into a map, then decode each property.

When a new property is added to a message, is it guaranteed to be added at the end?

No, there are no ordering guarantees.

When decoding a Typed, is the order of properties guaranteed?

When Pkl is encoded into binary, the order of properties on a Typed will follow the order as declared on the class.

When decoding a Dynamic, is the order of elements guaranteed?

The order of element object members will follow the same order as declared in Pkl.

[bioball]

You mean a compliant decoder of the Pkl language binding protocol? Would be good to clarify this in the spec.

A complaint decoder of messages in our message passing api

Is there a way to declare an element order other than 0, 1, 2, etc. in Pkl?

Not currently, but it isn't guaranteed to stay this way. If we ever implement element deletions, one possible design choice here is to have listings with holes.

Spec says that RemoteDependency.packageUri is optional

Thanks, this is an error!

MessagePackEncoder gets the map header wrong when encoding CreateEvaluatorRequest. I guess this code isn’t actually used.

It's exercised by our unit tests, but otherwise, we don't have a use-case for this right now.

Message encoding/decoding would be easier to implement and maintain if the message body was an array of nil-able values instead of a map of possibly absent values. In particular, this would make it unnecessary to have string literals for all property keys
count the number of present properties when writing the msgpack header first decode into a map.

Generally, MessagePack libraries can encode/decode these structs without needing to count entries. It's actually only on the Java side that this is low level.

In Go, for example, we use a struct with struct tags

In Swift, it's a Codable

It would be nice to have a Pkl file for the message schema. A commented Pkl file would make for a better schema spec than a mix of text and Pkl snippets.

We do have one, unless I'm missing something here? Our message schema is described using Pkl here: https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#messages

PS: I’m working on a Java language binding, which I think will be preferable to pkl-executor etc. in many cases. Also, Java 17’s records and sealed classes/interfaces enable better and more future-proof APIs.

Awesome! Yeah once we drop Java 11 support, we can support more modern Java features in pkl-codegen-java too.

[translatenix]

Not currently, but it isn't guaranteed to stay this way. If we ever implement element deletions, one possible design choice here is to have listings with holes.

Listings with holes sound very scary. Holes will break core assumptions that existing Pkl/Java/etc. code makes about listings.

If I'm not mistaken, even pkl-core APIs represent listings as java.util.Lists.

Not being able to rely on a contiguous ascending element index in the binary encoding also forces language bindings to internally represent elements as a map, which is quite undesirable. Note that interleaving of elements, properties, and entries isn't a problem--what matters is the relative order of elements.

It's exercised by our unit tests

Apparently this bug isn't covered by the tests.

We do have one, unless I'm missing something here? Our message schema is described using Pkl here: https://pkl-lang.org/main/current/bindings-specification/message-passing-api.html#messages

What I mean is that linking to an actual Pkl module, perhaps even one that can be viewed at https://pkl-lang.org/package-docs/ and in the IDE, would be even better than having a web page with embedded snippets of Pkl class bodies.

[translatenix]

According to the binary encoding spec, listings cannot have holes. They are encoded as arrays. Only Dynamic and Typed have this problem. For now I'll assume that Dynamic elements cannot have holes and Typed cannot have elements. AFAIK that's the current language behavior.

I think the binary encoding spec should offer the same guarantees as the language. Otherwise, language bindings will be forced to deal with hypothetical cases that might never happen and only complicate language binding APIs and implementations.

[bioball]

Hmmm.. yeah. We'd need to change that then if we have end up with listings with holes.

By the way: the rationale for "holes" in a listing is for if we introduce a syntax for "member deletion". For example:

class Person { name: String }

res1 = new Listing<Person> {
  new { name = "Leslie" }
  new { name = "Taylor" }
  new { name = "Jamie" }
}

res2 = (res1) {
  [1] { name = "Swift" }
}

res3 = (res2) {
  [0] = delete // imagined new syntax for element deletion
}

Without holes, [0] = delete will shift all elements up an index. It changes what [1] { name = "Swift" } amends (instead of s/Talyor/Swift/, it is s/Jamie/Swift/). It seems like behavior that would be pretty surprising, and error-prone for users.

Our thinking still TBD here. Your concerns around arrays with holes are valid, too.

[translatenix]

Perhaps you could support hiding elements instead of deleting them.
Then references wouldn't break, and the outside world could still be presented with contiguous elements starting from index 0.
Just one more option to think through.