.id files in Filecoin Spec

[yiannisbot]

As we are updating the Filecoin Specs site, we have been finding several outdated code chunks that mostly come from *.id interface files. We would like to understand how to best make use of this interface language, if it is useful to the engineering teams and the community at large, or if we should move to some other way of including and pointing to up-to-date code as we move forward.

Ultimately, if the filecoin teams want to make use of it going forward, we need to understand why it has not been kept up-to-date, as well as figure out convenient ways to keep it so.

[anorth]

It is not useful to the engineering teams.

[hugomrdias]

What about people outside the engineering teams ? new implementers, general users that want to understand how filecoin works ?

What about the go files generated from them that live (mostly in the spec repo), should we instead embed go files from the implementation repo ?

[jzimmerman]

As the author of .id, I'll try to add some color here. The purpose of the language was to extend Golang with a number of features that would facilitate concise, intuitive specs of protocols and data structures: most notably, sum types (aka. tagged unions) and programmatically-generated serialization/hashing of algebraic datatypes.

As it turned out, neither of these key features ended up being used at any significant scale. Since many mainstream languages, notably Golang itself, do not provide sum types, the decision as I understand it was to simply avoid using them when possible (and simulate them with explicitly tagged structs instead, when absolutely necessary). This in turn made the serialization/hashing task more straightforward, and reduced the additional benefit of the more flexible code generation approach proposed in the .id compiler.

So, in summary, I would distinguish between a few separate questions.

• Should the spec include code/pseudocode snippets of some form? I think the answer is pretty clearly yes; they often aid with clarity in presentation.

• Should those snippets be compilable? Opinions vary on this. My inclination is that this is helpful, as long as it's not a significant additional infrastructure burden.

• Should the language for those snippets be .id.? Probably not. As described above, the distinctive features of .id are not leveraged by the implementation, so it does not seem to justify maintaining the additional compiler toolchain. Golang is the predominant implementation language anyway, so that should be fine for the snippets in the spec.

[vmx]

• Should the language for those snippets be .id.? Probably not. As described above, the distinctive features of .id are not leveraged by the implementation, so it does not seem to justify maintaining the additional compiler toolchain. Golang is the predominant implementation language anyway, so that should be fine for the snippets in the spec.

For data structures a natural choice would be IPLD Schemas. They spec once had them, but they got removed at one point (some people still mourning).

[mikeal]

We’ve actually found it quite useful to have a clear distinction between logic/compute and the layout of data structures.

Since these are serialized and sent over networks there are implications that always go beyond how we are using them during the initial implementation. Using IPLD Schemas has been really helpful here, we’ve been able to have clear conversations about data structure layout in a language-agnostic way and in a DSL we all quickly have come to understand. IPLD Schemas also have the added benefit of not having language features that might allow you to paper over serialization issues and data model limitations which are all constraints that are in place for a reason and should be considered in the design of these layouts.

I don’t have a strong opinion on *.id or any other language or DSL for that matter, but I do think it would be useful to tease apart the design of the layout from the rest of what *.id is doing, and since we have tooling and expertise around IPLD Schemas it would make sense to use it for the layout.

[warpfork]

I'll mention a couple of quick things from my perspective while working on IPLD Schemas (... Mikeal already beat me to half of them!):

• I couldn't agree more than sum types are ridiculously useful and I want them to be something we can talk about clearly in protocol design!

• Getting sum types to work well in golang is also tricky I certainly agree :( (We're implementing at least two different forms of how we generate them in golang codegen, depending on which performance tradeoffs you want to make with them...)

• In general with IPLD Schemas, we've been very careful to define the type system in its own terms, not allowing golang to color it significantly (even as I implement the first draft of many things in golang). I'd hope this makes it all the more attractive for specifications with the explicit goal of being equally legible to readers in multiple languages. (Or at least equally illegible, so it puts everyone on equal footing and with equal interest in improving the docs paired with it!)

• We're committed to continued support of the IPLD Schema system, and growing it. We are fairly happy with our experiences using IPLD Schemas as a design language for other IPLD components, so this makes us confident in going further.

• It is not necessarily the case that we'll provide direct code generation support based on IPLD Schemas, either. Yet. (The initial emphasis on language-agnosticism also meant that development has taken longer; c'est la vie.) We do hope to offer this soon, though. And we will offer schema linting and vetting tools which are freestanding, independent of any use of codegen (and certainly independent of having, e.g., a golang compiler involved). Mostly these exist in javascript today, but more of these things built in golang should also become available in the near future.

tl;dr If you'd love to use IPLD Schemas in your specs, we'd love to support you.

[yiannisbot]

Just to provide a quick clarification from the specs project side on the following:

Should those snippets be compilable? Opinions vary on this. My inclination is that this is helpful, as long as it's not a significant additional infrastructure burden.

The goal is that it should be able to eventually do (as per the initial vision for the site). However, for the time being and in order to avoid extra work that does not seem to be necessary at this specific point in time, we have decided to hibernate this goal. None of the updates done on the website right now is prohibitive to switching back to compilable code and the decision is deferred for later.

That said, any decision taken on .id or not .id (and the tradeoffs it carries with it) should be sustainable in the longer term.

[yiannisbot]

TL:DR is that there is consensus to move away from .id, as far as I can gather.

Some more thoughts:

If you'd love to use IPLD Schemas in your specs, we'd love to support you.

I think that's great and we should move towards that direction in the future given the expertise and tools exist. But it does not fall within the short-term targets of the specs project to FIL launch.

• Should the spec include code/pseudocode snippets of some form? I think the answer is pretty clearly yes; they often aid with clarity in presentation.

Absolutely.

• Should those snippets be compilable? Opinions vary on this. My inclination is that this is helpful, as long as it's not a significant additional infrastructure burden.

Agreed. As mentioned above, this is the long-term vision, but not something we should target in the short term.

• Should the language for those snippets be .id.? Probably not. As described above, the distinctive features of .id are not leveraged by the implementation, so it does not seem to justify maintaining the additional compiler toolchain. Golang is the predominant implementation language anyway, so that should be fine for the snippets in the spec.

Agreed.

--

Thanks everyone for the feedback. Closing this issue. Discussion on "code in spec" continues here: #1065