encoding/json: parser ignores the case of member names

[cyberphone]

1. What version of Go are you using? 5.3
2. What operating system and processor architecture are you using? amd64,windows
3. What did you do?
   Read this: https://mailarchive.ietf.org/arch/msg/json/Ju-bwuRv-bq9IuOGzwqlV3aU9XE
4. What did you expect to see?
   ...
5. What did you see instead?
   ...

[bradfitz]

Playgroud link: http://play.golang.org/p/9j0ome9HqK

@rsc, @adg, thoughts? This surprised me. I thought we only did the case insensitive thing when there was no struct tag.

But from the ietf link above:

is looks quite diabolical for security as it is trivial to create valid JSON values that will be interpreted differently by different implementations.

Related to "differently by different implementations", we permit JSON keys twice: http://play.golang.org/p/lPgEj1T6Zk (two "alg"). That probably differs between implementations in either its output or whether it's even successful.

[rsc]

This has been the behavior at least as far back as Go 1.2 (I can't run
earlier on my Mac).
The docs also seem to state quite clearly that this is what happens:

To unmarshal JSON into a struct, Unmarshal matches incoming object keys to
the keys used by Marshal (either the struct field name or its tag),
preferring an exact match but also accepting a case-insensitive match.
Unmarshal will only set exported fields of the struct.

I understand there are security implications if JSON is used in security
contexts, and I was a little surprised too, but the docs are very clear. I
doubt that encoding/json's behavior should be dictated by security
concerns. FWIW, I don't believe we should go out of our way to reject
repeated fields either, especially not now. I might go so far as to argue
that using JSON in security standards is a mistake anyway, but I won't do
that here.

In any event it's too late to change the defaults. If we want to support
the security use case, maybe we could have a UseStrictNames method on
Decoder (like UseNumber).

[bradfitz]

UseStrictNames SGTM.

[cyberphone]

I would consider addressing a bunch of related issue as an option:
#14749
#14135

[rsc]

The other issues are unrelated and shouldn't be mixed in here. For one thing, we were talking about a decoding problem and #14749 is an encoding problem.

[manger]

UseStrictNames does look like a decent way to add case-sensitive decoding support in a backward compatible way.

However, while that would make case-sensitive decodes possible, it wouldn't help make this safer mode common or the default.

How about also defining a "strict" field tag option (c.f. "omitempty")? That should allow types to be safely used via newDecoder, or Unmarshall, or when the decoding is within a library. You can define the safety in the type, without finding every place it might be decoded.

[cespare]

Shall I send a UseStrictNames CL? Should that apply strict names across the board, whether or not the user supplied a struct tag?

type Foo struct {
        Bar string
        Baz string `json:"baz"`
}

So that would only match exactly "Bar" and "baz".

[rsc]

@cespare If it's simple, then yes sure. But after a thought on #15314 I wonder if maybe the UseStrictNames method is the wrong approach and instead it should be a tag attribute on the field. Then it can be enforced by the author of the struct instead of the author of the unmarshal call. Specifically, something like json:"Foo,exactname".

[jessfraz]

Would it get super repetitive if the person had to set it on every tag name?

[smyrman]

I think the CL is interesting, but it would, from my perspective, be even more useful to explicitly define an option. It's useful to be able to set that a case-insensitive match be treated as an unknown field.

As a comment on #14750 (comment):

I would vote for allowing case-sensitive matching as a decoder option enabled through a method call. (UseStrictNames, DisableEqualFold, or some better name) over struct-tags.

I want to advocate for this solution not because it's a better solution, but because it's in line with the current design for DisallowUnknownFields and UseNumbers. Having one annoying -- but consistent -- API has a value; it means the same work-around can be used for all cases where a custom json.Unmarshaler implementation is needed.

The DisallowUnknownFields and UseNumbers are in my opinion within the same class of problems, and that puts some restrains on how a strictly case-sensitve parser should be implemented.

Some notes:

• Projects can easily define their own jsonDecoder(data []byte) json.Decoder helper-functions to reduce the boiler plate in json.Unmarshaler implementations.
• Making a new json.Unmarshaler interface that accepts passing along options (or better still, accept a pre-initialized (sub) json.Docoder with inherited options) is certainly possible, but it's a separate issue. It is not given that it's always preferred.

[smyrman]

PS! The Encoder/Decoder interfaces defined in #12001 (old issue) could serve as a way of passing along decoder option. Perhaps that problem can be discussed there, although the motivation here is different.

[icholy]

@rsc this issue seems to have died down and gone off-topic. Would it be appropriate to reboot this as a proposal for UseStrictNames?

[zigo101]

preferring an exact match but also accepting a case-insensitive match

The wording of the first half line is some misleading IMHO.
It contains negative information.
I recommend to remove this half line to match the current behavior.

[nemith]

Just learned about this bahavior and this screams unintended bugs (no matter how much documentation). I always assumed that explicit struct tags take precedence over field names but they seem to just be ignore if there i field that matched.

[krigga]

This breaks binance API parsing (see https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-ticker-streams)
The API returns objects that have many single-character properties, many of them are the same letter but in different cases (so you pretty much have half the object's properties confused by the parser), and as far as I understand, after 6 years of this issue being known, there is still no workaround aside from parsing to a map[string]interface{} and then parsing to the desired object
I don't mind implementing a decoder option or whatever, but it needs to be decided so that I or someone else can implement it

[Lz-Gustavo]

@krigga a different workaround wouldn't be to declare all possible attributes cases, even those not required by the application?

For example, considering Binance's ticker payload, even if you only wanted to parse the "a" atribute, it seems to me that you could avoid the mismatching with the "A" field by declaring both on your target structure and simply ignoring the unwanted value https://go.dev/play/p/1_71Bw5FBJz.

Of course, this is just a workaround for the issue. One shouldn't be required to parse unwanted atributes for the sole purpose of avoiding this behavior. On this same snippet, removing any of the struct fields (by commenting line 14 or 15) leaves the script susceptible to the mentioned awkward behavior, by parsing "a" into an atribute with an "A" field tag.

[james-d-elliott]

This fork deals with the issue and implements the original proposed change I believe; https://github.com/james-d-elliott/go-json

It's a 1:1 fork of go1.19.3 other than the additional method and usage of said method. If the original proposal ever gets traction and is accepted you're welcome to ping me and/or just use the code as-is/modified with/without crediting me.

[dsnet]

Hi all, we kicked off a discussion for a possible "encoding/json/v2" package that addresses the spirit of this proposal.

In v2, we propose that Unmarshal perform a case-sensitive match on the field name by default.

For flexibility, the v2 prototype provides the ability to alter this behavior with:

• json.Unmarshal(v, json.MatchCaseInsensitiveNames(false)) at the unmarshal-call level OR
• struct{ FooField T `json:",nocase"` } at the struct-field level

[disconnect3d]

Hi, we stumbled upon this issue at Trail of Bits during one of our security assessments. We found a critical vulnerability due to how two different parsers treated similar JSON keys with matching text but differing capitalization - one of which was a Go program. A bit off-topic, but this is also not the first time that the lack of specification on how to treat duplicate JSON keys caused a critical vuln, as per Parsing JSON is a Minefield 💣:

[Update 2017-11-18] A RCE vulnerability was found in CouchDB because two JSON parsers handle duplicate key differently. The same JSON object, when parsed in JavaScript, contains "roles": []', but when parsed in Erlang it contains "roles": ["_admin"].

Anyway, I am writing here to encourage you to fix this issue - and at least request that this feature be documented better. I’m requesting this because the latest - Go 1.23 - documentation for JSON.Unmarshal is still incorrect - or at least very unclear:

To unmarshal JSON into a struct, Unmarshal matches incoming object keys to the keys used by Marshal (either the struct field name or its tag), preferring an exact match but also accepting a case-insensitive match. By default, object keys which don't have a corresponding struct field are ignored (see Decoder.DisallowUnknownFields for an alternative).

The doc says it "prefers an exact key match"; however, the JSON.Unmarshal code, when decoding object keys, simply goes over all the keys and overwrites (through this line in d.literalStore) the value of an exact key match if a non-exact (case insensitive) key match is found later on. While this was shown in multiple cases already, this one - https://go.dev/play/p/BL8Sw_KEUz9 - shows both cases and confirms that it is always the last key which is used:

[...]
        # we always end up with "last" value
	deserialize([]byte(`{"someName": "first", "SOMEname": "last"}`))
	deserialize([]byte(`{"SOMEname": "first", "someName": "last"}`))

Now, it seems that different solutions were suggested here such as:

• Multiple suggestions to add other JSON decoder options such as "UseStrictNames" or "DisallowEqualFold".
• There was an attempt to clarify the documentation in 221117: encoding/json: clarify how we decode into structs - which seems that it wasn't accepted because documenting it could "accept" the behavior and make it harder to change in the future
• There was an attempt to change this behavior in 224079: encoding/json: skip inexact matches after exact matches
• There is a encoding/json/v2 discussion which seems to be an ongoing effort of creating a new API that would hopefully fix this. The discussion includes a go-json-experiment/json experiment implementation.

Having written all of this, I still believe we could at least improve the documentation in the current implementation to make this clear once and for all.

[dsnet]

Making the documentation more clear sounds good to me. At this point, I don't think we can change the v1 behavior.

The v2 package will be the path forward for fixing this as we're aware that this is a potential security vulnerability. For those interested, here's an explanation how duplicate names (and case-insensitive names by extension) can be exploited: https://youtu.be/avilmOcHKHE?feature=shared&t=1011

[ianlancetaylor]

@dsnet: What is your opinion today on https://go.dev/cl/221117 ?

[mvdan]

Yes, if we're not going to change the behavior in v1, I'd also document it. I'm still okay with my wording from four years ago, but happy to get another review.