proposal: encoding/json: add "inline" struct tag

[gopherbot]

by bjruth:

Discussion was conceived on golang-nuts:
https://groups.google.com/forum/#!topic/golang-nuts/bAgNll-EMkI to add support for a
flag that supports unmarshaling arbitary JSON into structs. A tag was announced for the
mgo/bson package here:
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/ZeP7PaXVDQo that
transparently stores data into a separate struct field denoted by "inline"
during an unmarshal and then merges that data back into the output during marshaling.

I believe this would be a very useful feature to add into the encoding/json package and
does not introduce any breaking changes in the Go 1.x series.

Edited by @dsnet on 2020-11-10 to fix formatting.

[adg]

Comment 1:

I think this is worthwhile. I have an unfinished proposal in my email drafts that is
basically exactly what Gustavo describes in his bson release announcement mail.
It should be simple enough that it could be done before the 1.2 cutoff. Any disagreement?

Labels changed: added priority-soon, go1.2, feature, removed priority-triage.

Owner changed to @adg.

Status changed to Accepted.

[adg]

Comment 2:

https://golang.org/cl/13180043

Status changed to Started.

[adg]

Comment 3:

This issue was closed by revision 466001d.

Status changed to Fixed.

[adg]

Comment 4:

Submitted the CL by mistake.

Status changed to Started.

[adg]

Comment 5:

This issue was closed by revision 27f4166.

Status changed to Fixed.

[adg]

Comment 6:

And that was the undo CL closing it again :/
New CL https://golang.org/cl/13324045/

Status changed to Started.

[robpike]

Comment 7:

Not going into 1.2.

Labels changed: removed go1.2.

[rsc]

Comment 8:

Not even sure this is worthwhile.

Status changed to Thinking.

[gopherbot]

Comment 9 by bjruth:

@rsc It's not that a manual approach doesn't work, but since the unmarshaler already
handles distinguishing known vs. unknown fields relative to the target interface, the
extra data can thrown into a "catch-all" field.

[rsc]

Comment 10:

Labels changed: added go1.3maybe.

[rsc]

Comment 11:

Labels changed: removed feature.

[rsc]

Comment 12:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 13:

Labels changed: added repo-main.

[lukescott]

Comment 14:

I really need this. I have a struct that has static and dynamic fields. The dynamic
fields (in a map) are chosen by the user and need to be inline with the static ones.

[extemporalgenome]

Comment 15:

If this is going to be similar to <labix.org/v2/mgo/bson>, ideally an "inline" map
or struct pointer would not be allocated (the field value would remain nil) until there
is something valid to stuff in it.

[gopherbot]

Comment 16 by relistan:

+1. I was pondering writing a similar thing and came across this. Essentially we query a
3rd party API that is not versioned and changes. Catching the leftovers would let us
better get a sense of what was skipped. At a minimum it's great for debugging.

[gopherbot]

Comment 17 by andrew@happyinspector.com:

+1 I have something where I have a some common fields (id, path, revision, etc) and I
use an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

[gopherbot]

Comment 18 by andrew@happyinspector.com:

+1 I have something where I have some common fields (id, path, revision, etc) and I use
an anonymous struct that is embedded in other structs. However, I can't implement a
custom json marshal on the common struct as then all the other structs will only return
the output from the marshalled common fields.

[zelch]

I've read through the comments a few times, and I'm confused.

The ticket is still open, is this still an active proposal? Are we waiting for a better suggestion on syntax / naming?

[smoyer64]

At this point, I'd be happy to have ANY solution - the work-arounds are killers!

[zelch]

@dsnet Ping on my question above.

What is the current status of this? 9 years later, and this is still an issue for many of us.

I would be perfectly happy to work on the issue, but since I am not sure what the current blocking point is, I'm not sure where to go from here.

Requiring that the 'inline' struct tag have either a blank name, or one of '-' or '+', plus the 'any' tag, so something like ",any", "-,any", or even "+,any" seems like it would get past all of the issues regarding simply using - or + as the only indicator.

Using 'any' seems to make sense, as that is used by the XML encoder, but I'm happy with another keyword if that is preferred.

[n-bruno]

If I'm reading the thread correctly, this was implemented by adg earlier in the thread here but was never accepted. The tag added was overflow.

I think it's a feature worth adding as leveraging embedding (anonymous field) won't always work, as I've encountered recently with generics.

[zelch]

Hi @n-bruno.

As best as I can tell, that change was merged by accident, backed out... And then never revisited, at least as far as I can tell.

In my case, what I really want to do is embed a map of additional arbitrary JSON key/value pairs, which, of course, also isn't possible with embedding.

[zelch]

@dsnet Ping again?

[pohly]

+1 for adding explicit inlining, in whatever form.

I was trying to "concatenate" two structs into one where both structs are only known through some interfaces:

type objectWithKind struct {
	Object
	schema.ObjectKind
}

In this case, schema.ObjectKind is meant to provide values not stored already in Object. Kubernetes currently temporarily modifies Object while encoding it, but that is unexpected (should be treated as read-only) and has led to data races.

The approach above doesn't work at the moment because anonymous interface fields, in contrast to anonymous structs, are not inlined.

[dsnet]

There's a prototype implementation of inline but it can only handle struct, map[string]V, and json.RawMessage types.

@pohly, I don't know how to efficiently implement inlining of interface types.

For performance, the "json" package pseudo-statically analyzes a Go struct type to derive a table of how it's supposed to be marshaled. To avoid repeating this expensive work, it caches the computed result. This approach requires approximately O(N) memory, where N is total number of unique Go types.

Once you allow inlining of interface types, you run into combinatorial explosion of possible pseudo-types.

As a simple example, consider:

type S struct {
    A any `json:",inline"`
}

The S.A field is unbounded in the number of types it could represent.
Thus, we would need to compute and cache the marshal table of S[T1] where S.A happened to be T1,
and also for S[T2], S[T3], ..., S[Tn] for every type that could be used with S.A.
Then you have to consider that T1, T2, T3, ... Tn themselves may also have inlined interface types.

[pohly]

I see. So even if it could be made to work, for example by not caching such types, performance would be worse than it it is now. We care about performance here in Kubernetes, so this wouldn't help.

Ideally what we would need is some kind of "override" feature where the encoder is told to ignore a certain field and instead encode some other value. But that is a different problem than inlining and doesn't belong here.

[dsnet]

Hi all, we kicked off a discussion for a possible "encoding/json/v2" package that addresses the spirit of this proposal.
See the "inline" struct tag option under the "Struct tag options" section.

[Sora233]

I wonder why I can see json:",inline" everywhere, even if it never exists.

[mitar]

Because non-standard JSON libraries do support it.