encoding/json/v2 #63397

dsnet · 2023-10-05T17:14:54Z

dsnet
Oct 5, 2023
Collaborator

This is a discussion intended to lead to a formal proposal.

This was written with input from @mvdan, @johanbrandhorst, @rogpeppe, @chrishines, @rsc.

Background

The widely-used "encoding/json" package is over a decade old and has served the Go community well. Over time, we have learned much about what works well and what does not. Its ability to marshal from and unmarshal into native Go types, the ability to customize the representation of struct fields using Go struct tags, and the ability of Go types to customize their own representation has proven to be highly flexible.

However, many flaws and shortcomings have also been identified over time. Addressing each issue in isolation will likely lead to surprising behavior when non-orthogonal features interact poorly. This discussion aims to take a cohesive and comprehensive look at "json" and to propose a solution to support JSON in Go for the next decade and beyond.

Improvements may be delivered either by adding new functionality to the existing "json" package and/or by introducing a new major version of the package. To guide this decision, let us evaluate the existing "json" package in the following categories:

Missing functionality
API deficiencies
Performance limitations
Behavioral flaws

Missing functionality

There are quite a number of open proposals, where the most prominent feature requests are:

The ability to specify custom formatting of time.Time (#21990)
The ability to omit specific Go values when marshaling (#11939, #22480, #50480, #29310, #52803, #45669)
The ability to marshal nil Go slices and maps as empty JSON arrays and objects (#37711, #27589)
The ability to inline Go types without using Go embedding (#6213)

Most feature requests could be added to the existing "json" package in a backwards compatible way.

API deficiencies

The API can be sharp or restrictive:

There is no easy way to correctly unmarshal from an io.Reader. Users often write json.NewDecoder(r).Decode(v), which is incorrect since it does not reject trailing junk at the end of the payload (#36225).
Options can be set on the Encoder and Decoder types, but cannot be used with the Marshal and Unmarshal functions. Similarly, types implementing the Marshaler and Unmarshaler interfaces cannot make use of the options. There is no way to plumb options down the call stack (#41144).
The functions Compact, Indent, and HTMLEscape write to a bytes.Buffer instead of something more flexible like a []byte or io.Writer. This limits the usability of those functions.

These deficiencies could be fixed by introducing new API to the existing "json" package in a backwards compatible way at the cost of introducing multiple different ways of accomplishing the same tasks in the same package.

Performance limitations

The performance of the standard "json" package leaves much to be desired. Setting aside internal implementation details, there are externally visible APIs and behaviors that fundamentally limit performance:

MarshalJSON: The MarshalJSON interface method forces the implementation to allocate the returned []byte. Also, the semantics require that the "json" package parse the result both to verify that it is valid JSON and also to reformat it to match the specified indentation.
UnmarshalJSON: The UnmarshalJSON interface method requires that a complete JSON value be provided (without any trailing data). This forces the "json" package to parse the JSON value to be unmarshaled in its entirety to determine when it ends before it can call UnmarshalJSON. Afterwards, the UnmarshalJSON method itself must parse the provided JSON value again. If the UnmarshalJSON implementation recursively calls Unmarshal, this leads to quadratic behavior. As an example, this is the source of dramatic performance degradation when unmarshaling into spec.Swagger (kubernetes/kube-openapi#315).
Encoder.WriteToken: There is no streaming encoder API. A proposal has been accepted but not implemented (#40127). The proposed API symmetrically matches Decoder.Token, but suffers from the same performance problems (see next point).
Decoder.Token: The Token type is an interface, which can hold one of multiple types: Delim, bool, float64, Number, string, or nil. This unfortunately allocates frequently when boxing a number or string within the Token interface type (#40128).
Lack of streaming: Even though the Encoder.Encode and Decoder.Decode methods operate on an io.Writer or io.Reader, they buffer the entire JSON value in memory. This hurts performance since it requires a second pass through the JSON. In theory, only the largest JSON token (i.e., a JSON string or number) should ever need to be buffered (#33714, #7872, #11046).

Limitations 1 and 2 can be resolved by defining new interface methods that operate on a streaming encoder or decoder.

However, type-defined streaming methods are blocked on limitation 3 and 4, which requires having an efficient, streaming encoder and decoder API (#40127, #40128).

Even if an efficient streaming API is provided, the "json" package itself would still be constrained by limitation 5, where it does not operate in a truly streaming manner under the hood (#33714, #7872, #11046).

The "json" package should operate in a truly streaming manner by default when writing to or reading from an io.Writer or io.Reader. Buffering the entire JSON value defeats the point of using an io.Reader or io.Writer. Use cases that want to avoid outputting JSON in the event of an error should call Marshal instead and only write the output if the error is nil. Unfortunately, the "json" package cannot switch to streaming by default since this would be a breaking behavioral change, suggesting that a v2 "json" package is needed to accomplish this goal.

Behavioral flaws

Various behavioral flaws have been identified with the "json" package:

Improper handling of JSON syntax: Over the years, JSON has seen increased amounts of standardization (RFC 4627, RFC 7159, RFC 7493, and RFC 8259) in order for JSON-based protocols to properly communicate. Generally speaking, the specifications have gotten more strict over time since loose guarantees lead to implementations disagreeing about the meaning of a particular JSON value.
- The "json" package currently allows invalid UTF-8, while the latest internet standard (RFC 8259) requires valid UTF-8. The default behavior should at least be compliant with RFC 8259, which would require that the presence of invalid UTF-8 to be rejected as an error.
- The "json" package currently allows for duplicate object member names. RFC 8259 specifies that duplicate object names result in unspecified behavior (e.g., an implementation may take the first value, last value, ignore it, reject it, etc.). Fundamentally, the presence of a duplicate object name results in a JSON value without any universally agreed upon semantic (#43664). This could be exploited by attackers in security applications and has been exploited in practice with severe consequences. The default behavior should err on the side of safety and reject duplicate names as recommended by RFC 7493.
While the default behavior should be more strict, we should also provide an option for backwards compatibility to opt-in to the prior behavior of allowing invalid UTF-8 and/or allowing duplicate names.
Case-insensitive unmarshaling: When unmarshaling, JSON object names are paired with Go struct field names using a case-insensitive match (#14750). This is a surprising default, a potential security vulnerability, and a performance limitation. It may be a security vulnerability when an attacker provides an alternate encoding that a security tool does not know to check for. It is also a performance limitation since matching upon a case-insensitive name cannot be performed using a trivial Go map lookup.
Inconsistent calling of type-defined methods: Due to "json" and its use of Go reflection, the MarshalJSON and UnmarshalJSON methods cannot be called if the underlying value is not addressable (#22967, #27722, #33993, #55890). This is surprising to users when their declared MarshalJSON and UnmarshalJSON methods are not called when the underlying Go value was retrieved through a Go map, interface, or another non-addressable value. The "json" package should consistently and always call the user-defined methods regardless of addressability. As an implementation detail, non-addressable values can always be made addressable by temporarily boxing them on the heap. This could arguably be considered a bug and be fixed in the current "json" package. However, previous attempts at fixing this resulted in the changes being reverted because it broke too many targets implicitly depending on the inconsistent calling behavior.
Inconsistent merge semantics: When unmarshaling into a non-empty Go value, the behavior is inconsistent about whether it clears the target, resets but reuses the target memory, and/or whether it merges into the target (#27172, #31924, #26946). Most oddly, when unmarshaling into a non-nil Go slice, the unused elements between the length and capacity are merged into without being zeroed first (#21092). The merge semantics of "json" came about organically without much thought given to a systematic approach to merging, leading to fragmented and inconsistent behavior.
Inconsistent error values: There are three classes of errors that can occur when handling JSON:
- Syntactic error: The input does not match the JSON grammar (e.g., an improperly escaped JSON string).
- Semantic error: The input is valid JSON, but there is a type-mismatch between the JSON value and the Go value (e.g., unmarshaling a JSON bool into a Go struct).
- I/O error: A failure occurred writing to or reading from an io.Writer or io.Reader. This class of errors never occurs when marshaling to or unmarshaling from a []byte.
The "json" package is currently inconsistent about whether it returns structured or unstructured errors. It is currently impossible to reliably detect each class of error.

These behavioral flaws of "json" cannot be changed without being a breaking change. Options could be added to specify different behavior, but that would be unfortunate since the desired behavior is not the default behavior. Changing the default behavior suggests the need for a v2 "json" package.

Proposal

The analysis above suggests that a new major version of the "json" package is necessary and worthwhile. In this section, we propose a rough draft of what a new major version could look like. Henceforth, we will refer to the existing "encoding/json" package as v1, and a hypothetical new major version as v2. This is a draft proposal as the proposed API and behavior is subject to change based on community discussion.

Goals

Let us define some goals for v2:

Mostly backwards compatible: If possible, v2 should aim to be mostly compatible with v1 in terms of both API and default behavior to ease migration. For example, the Marshal and Unmarshal functions are the most widely used declarations in v1. It is sensible for equivalent functionality in v2 to be named the same and have mostly the same signature. Behaviorally, we should aim for 95% to 99% backwards compatibility. We do not aim for 100% compatibility since we want the freedom to break certain behaviors that are now considered to have been a mistake.
More correct: JSON standardization has become increasingly more strict over time due to interoperability issues. The default serialization should prioritize correctness.
More performant: JSON serialization is widely used and performance gains translate to real-world resource savings. However, performance is secondary to correctness. For example, rejecting duplicate object names will hurt performance, but is the more correct behavior to have.
More flexible: We should aim to provide the most flexible features that address most usages. We do not want to overfit v2 to handle every possible use case. The provided features should be orthogonal in nature such that any combination of features results in as few surprising edge cases as possible.
Easy to use (hard to misuse): The API should aim to make the common case easy and the less common case at least possible. The API should avoid behavior that goes contrary to user expectation, which may result in subtle bugs.
Avoid unsafe: JSON serialization is used by many internet-facing Go services. It is paramount that untrusted JSON inputs cannot result in memory corruption. Consequently, standard library packages generally avoid the use of package "unsafe" even if it could provide a performance boost. We aim to preserve this property.
- There are many community forks or reimplementations of v1 "json". While they provide impressive performance gains, they cannot be adopted into the standard library on the basis of their extensive use of package "unsafe". The 2021 Go Developer Survey shows that the assurance of reliability and security is a higher priority than CPU or memory performance.

Overview

JSON serialization can be broken down into two primary components:

syntactic functionality that is concerned with processing JSON based on its grammar, and
semantic functionality that determines the meaning of JSON values as Go values and vice-versa.

We use the terms "encode" and "decode" to describe syntactic functionality and the terms "marshal" and "unmarshal" to describe semantic functionality.

We aim to provide a clear distinction between functionality that is purely concerned with encoding versus that of marshaling. For example, it should be possible to encode a stream of JSON tokens without needing to marshal a concrete Go value representing them. Similarly, it should be possible to decode a stream of JSON tokens without needing to unmarshal them into a concrete Go value.

In v2, we propose that there be two packages: "jsontext" and "json". The "jsontext" package is concerned with syntactic functionality, while the "json" package is concerned with semantic functionality. The "json" package will be implemented in terms of the "jsontext" package. In order for "json" to marshal from and unmarshal into arbitrary Go values, it must have a dependency on the "reflect" package. In contrast, the "jsontext" package will have a relatively light dependency tree and be suitable for applications (e.g., TinyGo, GopherJS, WASI, etc.) where binary bloat is a concern.

This diagram provides a high-level overview of the v2 API. Purple blocks represent types, while blue blocks represent functions or methods. The direction of the arrows represent the approximate flow of data. The bottom half (as implemented by the "jsontext" package) of the diagram contains functionality that is only concerned with syntax, while the upper half (as implemented by the "json" package) contains functionality that assigns semantic meaning to syntactic data handled by the bottom half.

The "jsontext" package

The jsontext package provides functionality to process JSON purely according to the grammar. This package will have a small dependency tree such that it results in minimal binary bloat. Most notably, it does not depend on Go reflection.

Overview

The basic API consists of the following:

package jsontext // "encoding/json/jsontext"

type Encoder struct { /* no exported fields */ }
func NewEncoder(io.Writer, ...Options) *Encoder
func (*Encoder) WriteToken(Token) error
func (*Encoder) WriteValue(Value) error

type Decoder struct { /* no exported fields */ }
func NewDecoder(io.Reader, ...Options) *Decoder
func (*Decoder) PeekKind() Kind
func (*Decoder) ReadToken() (Token, error)
func (*Decoder) ReadValue() (Value, error)
func (*Decoder) SkipValue() error

type Kind byte
type Token struct { /* no exported fields */ }
func (Token) Kind() Kind
type Value []byte
func (Value) Kind() Kind

Values and Tokens

The primary data types for interacting with JSON are Kind, Value, and Token.

The Kind is an enumeration that describes the kind of a value or token.

// Kind represents each possible JSON token kind with a single byte,
// which is the first byte of that kind's grammar:
//   - 'n': null
//   - 'f': false
//   - 't': true
//   - '"': string
//   - '0': number
//   - '{': object start
//   - '}': object end
//   - '[': array start
//   - ']': array end
type Kind byte
func (k Kind) String() string

A Value is the raw representation of a single JSON value, which can represent entire array or object values. It is analogous to the v1 RawMessage type.

type Value []byte

func (v Value) Clone() Value
func (v Value) String() string
func (v Value) IsValid() bool
func (v *Value) Compact() error
func (v *Value) Indent(prefix, indent string) error
func (v *Value) Canonicalize() error
func (v Value) MarshalJSON() ([]byte, error)
func (v *Value) UnmarshalJSON(b []byte) error
func (v Value) Kind() Kind // never ']' or '}' if valid

The Compact and Indent methods operate similar to the v1 Compact and Indent function.
The Canonicalize method canonicalizes the JSON value according to the JSON Canonicalization Scheme as defined in RFC 8785.

A Token represents a lexical JSON token, which cannot represent entire array or object values. It is analogous to the v1 Token type, but is designed to be allocation-free by being an opaque struct type.

type Token struct { /* no exported fields */ }

var (
	Null  Token = rawToken("null")
	False Token = rawToken("false")
	True  Token = rawToken("true")

	ObjectStart Token = rawToken("{")
	ObjectEnd   Token = rawToken("}")
	ArrayStart  Token = rawToken("[")
	ArrayEnd    Token = rawToken("]")
)

func Bool(b bool) Token
func Int(n int64) Token
func Uint(n uint64) Token
func Float(n float64) Token
func String(s string) Token
func (t Token) Clone() Token
func (t Token) Bool() bool
func (t Token) Int() int64
func (t Token) Uint() uint64
func (t Token) Float() float64
func (t Token) String() string
func (t Token) Kind() Kind

Encoder and Decoder

The Encoder and Decoder types provide the functionality for encoding to or decoding from an io.Writer or an io.Reader. An Encoder or Decoder can be constructed withNewEncoder or NewDecoder using default options.

The Encoder is a streaming encoder from raw JSON tokens and values. It is used to write a stream of top-level JSON values, each terminated with a newline character.

type Encoder struct { /* no exported fields */ }

func (e *Encoder) Reset(w io.Writer, opts ...Options)

// WriteToken writes the next token and advances the internal write offset.
// The provided token must be consistent with the JSON grammar.
func (e *Encoder) WriteToken(t Token) error

// WriteValue writes the next raw value and advances the internal write offset.
// The provided value must be consistent with the JSON grammar.
func (e *Encoder) WriteValue(v Value) error

// UnusedBuffer returns a zero-length buffer with a possible non-zero capacity.
// This buffer is intended to be used to populate a Value
// being passed to an immediately succeeding WriteValue call.
//
// Example usage:
//
//	b := d.UnusedBuffer()
//	b = append(b, '"')
//	b = appendString(b, v) // append the string formatting of v
//	b = append(b, '"')
//	... := d.WriteValue(b)
func (e *Encoder) UnusedBuffer() []byte

// OutputOffset returns the current output byte offset, which is the location
// of the next byte immediately after the most recently written token or value.
func (e *Encoder) OutputOffset() int64

The Decoder is a streaming decoder for raw JSON tokens and values. It is used to read a stream of top-level JSON values, each separated by optional whitespace characters.

type Decoder struct { /* no exported fields */ }

func (d *Decoder) Reset(r io.Reader, opts ...Options)

// PeekKind returns the kind of the token that would be returned by ReadToken.
// It does not advance the read offset.
func (d *Decoder) PeekKind() Kind

// ReadToken reads the next Token, advancing the read offset.
// The returned token is only valid until the next Peek, Read, or Skip call.
// It returns io.EOF if there are no more tokens.
func (d *Decoder) ReadToken() (Token, error)

// ReadValue returns the next raw JSON value, advancing the read offset.
// The returned value is only valid until the next Peek, Read, or Skip call
// and may not be mutated while the Decoder remains in use.
// It returns io.EOF if there are no more values.
func (d *Decoder) ReadValue() (Value, error)

// SkipValue is equivalent to calling ReadValue and discarding the result except
// that memory is not wasted trying to hold the entire value.
func (d *Decoder) SkipValue() error

// UnreadBuffer returns the data remaining in the unread buffer.
// The returned buffer must not be mutated while Decoder continues to be used.
// The buffer contents are valid until the next Peek, Read, or Skip call.
func (d *Decoder) UnreadBuffer() []byte

// InputOffset returns the current input byte offset, which is the location
// of the next byte immediately after the most recently returned token or value.
func (d *Decoder) InputOffset() int64

Some methods common to both Encoder and Decoder report information about the current automaton state.

// StackDepth returns the depth of the state machine.
// Each level on the stack represents a nested JSON object or array.
// It is incremented whenever an ObjectStart or ArrayStart token is encountered
// and decremented whenever an ObjectEnd or ArrayEnd token is encountered.
// The depth is zero-indexed, where zero represents the top-level JSON value.
func (e *Encoder) StackDepth() int
func (d *Decoder) StackDepth() int

// StackIndex returns information about the specified stack level.
// It must be a number between 0 and StackDepth, inclusive.
// For each level, it reports the kind:
//
//   - 0 for a level of zero,
//   - '{' for a level representing a JSON object, and
//   - '[' for a level representing a JSON array.
//
// It also reports the length so far of that JSON object or array.
// Each name and value in a JSON object is counted separately,
// so the effective number of members would be half the length.
// A complete JSON object must have an even length.
func (e *Encoder) StackIndex(i int) (Kind, int)
func (d *Decoder) StackIndex(i int) (Kind, int)

// StackPointer returns a JSON Pointer (RFC 6901) to the most recently handled value.
// Object names are only present if AllowDuplicateNames is false, otherwise
// object members are represented using their index within the object.
func (e *Encoder) StackPointer() string
func (d *Decoder) StackPointer() string

Options

The behavior of Encoder and Decoder may be altered by passing options to NewEncoder and NewDecoder, which take in a variadic list of options.

type Options = jsonopts.Options

// AllowDuplicateNames specifies that JSON objects may contain
// duplicate member names.
func AllowDuplicateNames(v bool) Options // affects encode and decode

// AllowInvalidUTF8 specifies that JSON strings may contain invalid UTF-8,
// which will be mangled as the Unicode replacement character, U+FFFD.
func AllowInvalidUTF8(v bool) Options // affects encode and decode

// EscapeForHTML specifies that '<', '>', and '&' characters within JSON strings
// should be escaped as a hexadecimal Unicode codepoint (e.g., \u003c)
// so that the output is safe to embed within HTML.
func EscapeForHTML(v bool) Options // affects encode only

// EscapeForJS specifies that U+2028 and U+2029 characters within JSON strings
// should be escaped as a hexadecimal Unicode codepoint (e.g., \u2028)
// so that the output is valid to embed within JavaScript.
// See RFC 8259, section 12.
func EscapeForJS(v bool) Options // affects encode only

// WithIndent specifies that the encoder should emit multiline output
// where each element in a JSON object or array begins on a new, indented line
// beginning with the indent prefix (see WithIndentPrefix) followed by
// one or more copies of indent according to the nesting depth.
func WithIndent(indent string) Options // affects encode only

// WithIndentPrefix specifies that the encoder should emit multiline output
// where each element in a JSON object or array begins on a new, indented line
// beginning with the indent prefix followed by
// one or more copies of indent (see WithIndent) according to the nesting depth.
func WithIndentPrefix(prefix string) Options // affects encode only

// Expand specifies that the JSON output should be expanded, where
// every JSON object member or JSON array element appears on a new, indented line
// according to the nesting depth.
// If an indent is not already specified, then it defaults to using "\t".
func Expand(v bool) Options // affects encode only

The Options type is a type alias to an internal type that is an interface type with no exported methods. It is used simply as a marker type for options declared in the "json" and "jsontext" package.

Latter option specified in the variadic list passed to NewEncoder and NewDecoder takes precedence over prior option values. For example, NewEncoder(AllowInvalidUTF8(false), AllowInvalidUTF8(true)) results in AllowInvalidUTF8(true) taking precedence.

Options that do not affect the operation in question are ignored. For example, passing Expand to NewDecoder does nothing.

The WithIndent and WithIndentPrefix flags configure the appearance of whitespace in the output. Their semantics are identical to the v1 Encoder.SetIndent method.

Errors

Errors due to non-compliance with the JSON grammar are reported as SyntacticError.

type SyntacticError struct {
	// ByteOffset indicates that an error occurred after this byte offset.
	ByteOffset int64
	// JSONPointer indicates that an error occurred within this JSON value
	// as indicated using the JSON Pointer notation (see RFC 6901).
	JSONPointer string
	// Err is the underlying error.
	Err error // always non-nil
}
func (e *SyntacticError) Error() string
func (e *SyntacticError) Unwrap() error

Errors due to I/O are returned as an opaque error that unwrap to the original error returned by the failing io.Reader.Read or io.Writer.Write call.

The v2 "json" package

The v2 "json" package provides functionality to marshal or unmarshal JSON data from or into Go value types. This package depends on "jsontext" to process JSON text and the "reflect" package to dynamically introspect Go values at runtime.

Overview

The basic API consists of the following:

package json // "encoding/json/v2"

func Marshal(in any, opts ...Options) (out []byte, err error)
func MarshalWrite(out io.Writer, in any, opts ...Options) error
func MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error

func Unmarshal(in []byte, out any, opts ...Options) error
func UnmarshalRead(in io.Reader, out any, opts ...Options) error
func UnmarshalDecode(in *jsontext.Decoder, out any, opts ...Options) error

The Marshal and Unmarshal functions mostly match the signature of the same functions in v1, however their behavior differs.

The MarshalWrite and UnmarshalRead functions are equivalent functionality that operate on an io.Writer and io.Reader instead of []byte. The UnmarshalRead function consumes the entire input until io.EOF and reports an error if any invalid tokens appear after the end of the JSON value (#36225).

The MarshalEncode and UnmarshalDecode functions are equivalent functionality that operate on an *jsontext.Encoder and *jsontext.Decoder instead of []byte.

Default behavior

The marshal and unmarshal logic in v2 is mostly identical to v1 with following changes:

See here for details about every change.

Every behavior change will be configurable through options, which will be a critical part of how we achieve v1-to-v2 interoperability.
See here for more discussion.

Struct tag options

Similar to v1, v2 also supports customized representation of Go struct fields through the use of struct tags. As before, the json tag will be used. The following tag options are supported:

omitzero: When marshaling, the "omitzero" option specifies that the struct field should be omitted if the field value is zero, as determined by the "IsZero() bool" method, if present, otherwise based on whether the field is the zero Go value (per reflect.Value.IsZero). This option has no effect when unmarshaling. (example)
- New in v2. The inability to omit an empty struct is a frequently cited issue in v1. This feature is intended to provide a general way to accomplish that goal (#11939, #22480, #50480, #29310, #52803, #45669).
omitempty: When marshaling, the "omitempty" option specifies that the struct field should be omitted if the field value would have been encoded as a JSON null, empty string, empty object, or empty array. This option has no effect when unmarshaling. (example)
- Changed in v2. In v1, the "omitempty" option was narrowly defined as only omitting a field if it is a Go false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string. In v2, it has been redefined in terms of the JSON type system, rather than the Go type system. They are practically equivalent except for Go bools and numbers, for which the "omitzero" option can be used instead (more discussion).
string: The "string" option specifies that StringifyNumbers be set when marshaling or unmarshaling a struct field value. This causes numeric types to be encoded as a JSON number within a JSON string, and to be decoded from either a JSON number or a JSON string containing a JSON number. This extra level of encoding is often necessary since many JSON parsers cannot precisely represent 64-bit integers.
- Changed in v2. In v1, the "string" option applied to certain types where use of a JSON string did not make sense (e.g., a bool) and could not be applied recursively (e.g., a slice of integers). In v2, this feature only applies to numeric types and applies recursively.
nocase: When unmarshaling, the "nocase" option specifies that if the JSON object name does not exactly match the JSON name for any of the struct fields, then it attempts to match the struct field using a case-insensitive match that also ignores dashes and underscores. (example)
- New in v2. Since v2 no longer performs a case-insensitive match of JSON object names, this option provides a means to opt-into the v1-like behavior. However, the case-insensitive match is altered relative to v1 in that it also ignores dashes and underscores. This makes the feature more broadly useful for JSON objects with different naming conventions to be unmarshaled. For example, "fooBar", "FOO_BAR", or "foo-bar" will all match with a field named "FooBar".
inline: The "inline" option specifies that the JSON object representation of this field is to be promoted as if it were specified in the parent struct. It is the JSON equivalent of Go struct embedding. A Go embedded field is implicitly inlined unless an explicit JSON name is specified. The inlined field must be a Go struct that does not implement Marshaler or Unmarshaler. Inlined fields of type jsontext.Value and map[string]T are called “inlined fallbacks”, as they can represent all possible JSON object members not directly handled by the parent struct. Only one inlined fallback field may be specified in a struct, while many non-fallback fields may be specified. This option must not be specified with any other tag option. (example)
- New in v2. Inlining is an explicit way to embed a JSON object within another JSON object without relying on Go struct embedding. The feature is capable of inlining Go maps and jsontext.Value (#6213).
unknown: The "unknown" option is a specialized variant of the inlined fallback to indicate that this Go struct field contains any number of “unknown” JSON object members. The field type must be a jsontext.Value, map[string]T. If DiscardUnknownMembers is specified when marshaling, the contents of this field are ignored. If RejectUnknownMembers is specified when unmarshaling, any unknown object members are rejected even if a field exists with the "unknown" option. This option must not be specified with any other tag option. (example)
- New in v2. The "inline" feature technically provides a way to preserve unknown member (#22533). However, the "inline" feature alone does not semantically tell us whether this field is meant to store unknown members. The "unknown" option gives us this extra bit of information so that we can cooperate with options that affect unknown membership.
format: The "format" option specifies a format flag used to specialize the formatting of the field value. The option is a key-value pair specified as "format:value" where the value must be either a literal consisting of letters and numbers (e.g., "format:RFC3339") or a single-quoted string literal (e.g., "format:'2006-01-02'"). The interpretation of the format flag is determined by the struct field type. (example)
- New in v2. The "format" option provides a general way to customize formatting of arbitrary types.
- []byte and [N]byte types accept "format" values of either "base64", "base64url", "base32", "base32hex", "base16", or "hex", where it represents the binary bytes as a JSON string encoded using the specified format in RFC 4648. It may also be "array" to treat the slice or array as a JSON array of numbers. The "array" format exists for backwards compatibility since the default representation of an array of bytes now uses Base-64.
- float32 and float64 types accept a "format" value of "nonfinite", where NaN and infinity are represented as JSON strings.
- Slice types accept a "format" value of "emitnull" to marshal a nil slice as a JSON null instead of an empty JSON array. (more discussion).
- Map types accept a "format" value of "emitnull" to marshal a nil map as a JSON null instead of an empty JSON object. (more discussion).
- The time.Time type accepts a "format" value which may either be a Go identifier for one of the format constants (e.g., "RFC3339") or the format string itself to use with time.Time.Format or time.Parse (#21990). It can also be "unix", "unixmilli", "unixmicro", or "unixnano" to be represented as a decimal number reporting the number of seconds (or milliseconds, etc.) since the Unix epoch.
- The time.Duration type accepts a "format" value of "sec", "milli", "micro", or "nano" to represent it as the number of seconds (or milliseconds, etc.) formatted as a JSON number. This exists for backwards compatibility since the default representation now uses a string representation (e.g., "53.241s"). If the format is "base60", it is encoded as a JSON string using the "H:MM:SS.SSSSSSSSS" representation.

The "omitzero" and "omitempty" options are similar. The former is defined in terms of the Go type system, while the latter in terms of the JSON type system. Consequently they behave differently in some circumstances. For example, only a nil slice or map is omitted under "omitzero", while an empty slice or map is omitted under "omitempty" regardless of nilness. The "omitzero" option is useful for types with a well-defined zero value (e.g., netip.Addr) or have an IsZero method (e.g., time.Time).

Type-specified customization

Go types may customize their own JSON representation by implementing certain interfaces that the "json" package knows to look for:

type MarshalerV1 interface {
	MarshalJSON() ([]byte, error)
}
type MarshalerV2 interface {
	MarshalJSONV2(*jsontext.Encoder, Options) error
}

type UnmarshalerV1 interface {
	UnmarshalJSON([]byte) error
}
type UnmarshalerV2 interface {
	UnmarshalJSONV2(*jsontext.Decoder, Options) error
}

The v1 interfaces are supported in v2 to provide greater degrees of backward compatibility. If a type implements both v1 and v2 interfaces, the v2 variant takes precedence. The v2 interfaces operate in a purely streaming manner. This API can provide dramatic performance improvements. For example, switching from UnmarshalJSON to UnmarshalJSONV2 for spec.Swagger resulted in an ~40x performance improvement.

Caller-specified customization

In addition to Go types being able to specify their own JSON representation, the caller of the marshal or unmarshal functionality can also specify their own JSON representation for specific Go types (#5901). Caller-specified customization takes precedence over type-specified customization.

// SkipFunc may be returned by MarshalFuncV2 and UnmarshalFuncV2 functions.
// Any function that returns SkipFunc must not cause observable side effects
// on the provided Encoder or Decoder.
const SkipFunc = jsonError("skip function")

// Marshalers holds a list of functions that may override the marshal behavior
// of specific types. Populate WithMarshalers to use it.
// A nil *Marshalers is equivalent to an empty list.
type Marshalers struct { /* no exported fields */ }

// NewMarshalers constructs a flattened list of marshal functions.
// If multiple functions in the list are applicable for a value of a given type,
// then those earlier in the list take precedence over those that come later.
// If a function returns SkipFunc, then the next applicable function is called,
// otherwise the default marshaling behavior is used.
//
// For example:
//
//	m1 := NewMarshalers(f1, f2)
//	m2 := NewMarshalers(f0, m1, f3)     // equivalent to m3
//	m3 := NewMarshalers(f0, f1, f2, f3) // equivalent to m2
func NewMarshalers(ms ...*Marshalers) *Marshalers

// MarshalFuncV1 constructs a type-specific marshaler that
// specifies how to marshal values of type T.
func MarshalFuncV1[T any](fn func(T) ([]byte, error)) *Marshalers

// MarshalFuncV2 constructs a type-specific marshaler that
// specifies how to marshal values of type T.
// The function is always provided with a non-nil pointer value
// if T is an interface or pointer type.
func MarshalFuncV2[T any](fn func(*jsontext.Encoder, T, Options) error) *Marshalers

// Unmarshalers holds a list of functions that may override the unmarshal behavior
// of specific types. Populate WithUnmarshalers to use it.
// A nil *Unmarshalers is equivalent to an empty list.
type Unmarshalers struct { /* no exported fields */ }

// NewUnmarshalers constructs a flattened list of unmarshal functions.
// It operates in a similar manner as NewMarshalers.
func NewUnmarshalers(us ...*Unmarshalers) *Unmarshalers

// UnmarshalFuncV1 constructs a type-specific unmarshaler that
// specifies how to unmarshal values of type T.
func UnmarshalFuncV1[T any](fn func([]byte, T) error) *Unmarshalers

// UnmarshalFuncV2 constructs a type-specific unmarshaler that
// specifies how to unmarshal values of type T.
// T must be an unnamed pointer or an interface type.
// The function is always provided with a non-nil pointer value.
func UnmarshalFuncV2[T any](fn func(*jsontext.Decoder, T, Options) error) *Unmarshalers

The MarshalFuncV1 and UnmarshalFuncV1 functions can always be implemented in terms of the v2 variants, which calls into question their utility. There are several reasons for providing them:

To maintain symmetry and consistency with the method interfaces (which must provide both v1 and v2 variants).
To make it interoperate well with existing functionality that operate on the v1 signature. For example, to integrate the v2 "json" package with proper JSON serialization of protocol buffers, one could construct a type-specific marshaler using json.MarshalFuncV1(protojson.Marshal), where protojson.Marshal provides the JSON representation for all types that implement proto.Message (example).

Caller-specified customization is a powerful feature. For example:

It can be used to marshal Go errors (example).
It can be used to preserve the raw representation of JSON numbers (example). Note that v2 does not have the v1 RawNumber type.
It can be used to preserve the input offset of JSON values for error reporting purposes (example).

Options

Options may be specified that configure how marshal and unmarshal operates:

// Options configure Marshal, MarshalWrite, MarshalEncode,
// Unmarshal, UnmarshalRead, and UnmarshalDecode with specific features.
// Each function takes in a variadic list of options, where properties set
// in latter options override the value of previously set properties.
//
// Options represent either a singular option or a set of options.
// It can be functionally thought of as a Go map of option properties
// (even though the underlying implementation avoids Go maps for performance).
//
// The constructors (e.g., Deterministic) return a singular option value:
//	opt := Deterministic(true)
// which is analogous to creating a single entry map:
//	opt := Options{"Deterministic": true}
//
// JoinOptions composes multiple options values to together:
//	out := JoinOptions(opts...)
// which is analogous to making a new map and copying the options over:
//	out := make(Options)
//	for _, m := range opts {
//		for k, v := range m {
//			out[k] = v
//		}
//	}
//
// GetOption looks up the value of options parameter:
//	v, ok := GetOption(opts, Deterministic)
// which is analogous to a Go map lookup:
//	v, ok := opts["Deterministic"]
//
// There is a single Options type, which is used with both marshal and unmarshal.
// Options that do not affect a particular operation are ignored.
type Options = jsonopts.Options

// StringifyNumbers specifies that numeric Go types should be marshaled as
// a JSON string containing the equivalent JSON number value.
// When unmarshaling, numeric Go types can be parsed from either a JSON number
// or a JSON string containing the JSON number without any surrounding whitespace.
func StringifyNumbers(v bool) Options // affects marshal and unmarshal

// Deterministic specifies that the same input value will be serialized
// as the exact same output bytes. Different processes of
// the same program will serialize equal values to the same bytes,
// but different versions of the same program are not guaranteed
// to produce the exact same sequence of bytes.
func Deterministic(v bool) Options // affects marshal only

// FormatNilMapAsNull specifies that a nil Go map should marshal as a
// JSON null instead of the default representation as an empty JSON object.
func FormatNilMapAsNull(v bool) Options // affects marshal only

// FormatNilSliceAsNull specifies that a nil Go slice should marshal as a
// JSON null instead of the default representation as an empty JSON array
// (or an empty JSON string in the case of ~[]byte).
func FormatNilSliceAsNull(v bool) Options // affects marshal only

// MatchCaseInsensitiveNames specifies that JSON object members are matched
// against Go struct fields using a case-insensitive match of the name.
func MatchCaseInsensitiveNames(v bool) Options // affects marshal and unmarshal

// DiscardUnknownMembers specifies that marshaling should ignore any
// JSON object members stored in Go struct fields dedicated to storing
// unknown JSON object members.
func DiscardUnknownMembers(v bool) Options // affects marshal only

// RejectUnknownMembers specifies that unknown members should be rejected
// when unmarshaling a JSON object, regardless of whether there is a field
// to store unknown members.
func RejectUnknownMembers(v bool) Options // affects unmarshal only

// WithMarshalers specifies a list of type-specific marshalers to use,
// which can be used to override the default marshal behavior
// for values of particular types.
func WithMarshalers(v *Marshalers) Options // affects marshal only

// WithUnmarshalers specifies a list of type-specific unmarshalers to use,
// which can be used to override the default unmarshal behavior
// for values of particular types.
func WithUnmarshalers(v *Unmarshalers) Options // affects unmarshal only

// JoinOptions coalesces the provided list of options into a single Options.
// Properties set in latter options override the value of previously set properties.
func JoinOptions(srcs ...Options) Options

// GetOption returns the value stored in opts with the provided constructor,
// reporting whether the value is present.
func GetOption[T any](opts Options, constructor func(T) Options) (T, bool)

The Options type is a type alias to an internal type that is an interface type with no exported methods. It is used simply as a marker type for options declared in the "json" and "jsontext" package. This is exactly the same Options type as the one in the "jsontext" package.

The same Options type is used for both Marshal and Unmarshal as some options affect both operations.

The MarshalJSONV2, UnmarshalJSONV2, MarshalFuncV2, and UnmarshalFuncV2 methods and functions take in a singular Options value instead of a variadic list because the Options type can represent a set of options. The caller (which is the "json" package) can coalesce a list of options before calling the user-specified method or function. Being given a single Options value is more ergonomic for the user as there is only one options value to introspect with GetOption.

While the JoinOptions constructor technically removes the need for NewEncoder, NewDecoder, Marshal, and Unmarshal from taking in a variadic list of options, it is more ergonomic for it to be variadic as the user can more readily specify a list of options without needing to call JoinOptions first.

Errors

Errors due to the inability to correlate JSON data with Go data are reported as SemanticError.

type SemanticError struct {
	// ByteOffset indicates that an error occurred after this byte offset.
	ByteOffset int64
	// JSONPointer indicates that an error occurred within this JSON value
	// as indicated using the JSON Pointer notation (see RFC 6901).
	JSONPointer string

	// JSONKind is the JSON kind that could not be handled.
	JSONKind Kind // may be zero if unknown
	// GoType is the Go type that could not be handled.
	GoType reflect.Type // may be nil if unknown

	// Err is the underlying error.
	Err error // may be nil
}
func (e *SemanticError) Error() string
func (e *SemanticError) Unwrap() error

Experimental implementation

The draft proposal has been implemented by the github.com/go-json-experiment/json module.

Stability

We have confidence in the correctness and performance of the module as it has been used internally at Tailscale in various production services. However, the module is an experiment and breaking changes are expected to occur based on feedback in this discussion, it should not be depended upon by publicly available code, otherwise we can run into situations where large programs fail to build.

Consider the following situation:

Program P depends on modules A and B.
Module A depends on go-json-experiment/json@v0.5.0.
Module B depends on go-json-experiment/json@v0.8.0.
Let’s suppose a breaking change occurs between v0.5.0 and v0.8.0.
MVS dictates that v0.8.0 be selected to build program P.
However, the use of v0.8.0 breaks module A since it is using the API for v0.5.0, which is not compatible.

If open source code does use go-json-experiment, we recommend that use of it be guarded by a build tag or the entire module be forked and vendored as a dependency.

Performance

Due to a combination of both a more efficient implementation and also changes to the external API to better support performance, the experimental v2 implementation is generally as fast or slightly faster for marshaling and dramatically faster for unmarshaling.

See the benchmarks for results.

dsnet · 2023-10-05T17:16:25Z

dsnet
Oct 5, 2023
Collaborator Author

It is imperative that v1 and v2 interoperate well to provide a gradual migration from v1 to v2. Any code using v1 today must continue to function the same today and into the future. The key to v1-to-v2 interoperability lies in the API for composable options.

Across the "jsontext" packages and v2 and v1 "json" packages, we have:

package jsontext // "encoding/json/jsontext"

type Options = jsonopts.Options

func AllowDuplicateNames(v bool) Options     // affects encode and decode
func AllowInvalidUTF8(v bool) Options        // affects encode and decode
func EscapeForHTML(v bool) Options           // affects encode only
func EscapeForJS(v bool) Options             // affects encode only
func WithIndent(indent string) Options       // affects encode only
func WithIndentPrefix(prefix string) Options // affects encode only
func Expand(v bool) Options                  // affects encode only

package json // "encoding/json/v2"

type Options = jsonopts.Options

// DefaultOptionsV2 is the full set of all options that define v2 semantics.
func DefaultOptionsV2() Options

func StringifyNumbers(v bool) Options          // affects marshal and unmarshal
func Deterministic(v bool) Options             // affects marshal only
func FormatNilMapAsNull(v bool) Options        // affects marshal only
func FormatNilSliceAsNull(v bool) Options      // affects marshal only
func MatchCaseInsensitiveNames(v bool) Options // affects marshal and unmarshal
func DiscardUnknownMembers(v bool) Options     // affects marshal only
func RejectUnknownMembers(v bool) Options      // affects unmarshal only
func WithMarshalers(v *Marshalers) Options     // affects marshal only
func WithUnmarshalers(v *Unmarshalers) Options // affects unmarshal only

package json // "encoding/json"

type Options = jsonopts.Options

// DefaultOptionsV1 is the full set of all options that define v1 semantics.
func DefaultOptionsV1() Options

func FormatByteArrayAsArray(v bool) Options         // affects marshal and unmarshal
func FormatTimeDurationAsNanosecond(v bool) Options // affects marshal and unmarshal
func IgnoreStructErrors(v bool) Options             // affects marshal and unmarshal
func MatchCaseSensitiveDelimiter(v bool) Options    // affects marshal and unmarshal
func MergeWithLegacySemantics(v bool) Options       // affects unmarshal only
func OmitEmptyWithLegacyDefinition(v bool) Options  // affects marshal only
func RejectFloatOverflow(v bool) Options            // affects unmarshal only
func ReportLegacyErrorValues(v bool) Options        // affects marshal and unmarshal
func SkipUnaddressableMethods(v bool) Options       // affects marshal and unmarshal
func StringifyWithLegacySemantics(v bool) Options   // affects marshal and unmarshal
func UnmarshalArrayFromAnyLength(v bool) Options    // affects unmarshal only

For brevity, we will use jsonv1 to refer to v1 "encoding/json" and jsonv2 to refer to "encoding/json/v2".

There are several things to note:

The Options type across all three packages are identical, which implies that options declared in all three packages can be used with the jsonv2.Marshal and jsonv2.Unmarshal functions.
The jsonopts.Options type is declared in an internal package. It is an interface type without exported methods. It exists only to mark options declared in the v1 and v2 "json" package and the "jsontext" package. At present, we do not permit user-declared options, but that could be a future extension to the design.
The Options type represents either a singular option or a set of options. The jsonv1.DefaultOptionsV1 and jsonv2.DefaultOptionsV2 options represent the full set of all options that define default v1 or v2 behavior. The JoinOptions function combines multiple options together to form larger sets of options.
In the variadic listing of options that jsonv2.Marshal and jsonv2.Unmarshal accepts, latter options take precedence over prior options.
The exact list of options in "jsonv1" is still to-be-determined, but there will be an option for every behavior change relative to v2. Options that are reasonable to toggle will be declared in v2, while obscure behavior changes will have their options declared in the v1 package.

Thus, these options can be composed together to obtain behavior that is identical to v1, identical to v2, or anywhere in between. For example:

jsonv1.Marshal(v)
- uses default v1 semantics
jsonv2.Marshal(in, jsonv1.DefaultOptionsV1)
- semantically equivalent to jsonv1.Marshal
jsonv2.Marshal(in, jsonv1.DefaultOptionsV1, jsontext.AllowDuplicateNames(false))
- uses mostly v1 semantics, but opts into one v2-specific behaviors
jsonv2.Marshal(in, jsonv1.FormatByteArrayAsArray(true), jsonv1.IgnoreStructErrors(true))
- uses mostly v2 semantics, but opts into two v1-specific behaviors
jsonv2.Marshal(v, ..., jsonv2.DefaultOptionsV2)
- semantically equivalent to jsonv2.Marshal since jsonv2.DefaultOptionsV2 overrides any options specified earlier in the ...
jsonv2.Marshal(v)
- uses default v2 semantics

The implementation of jsonv1.Marshal will actually become jsonv2.Marshal(..., jsonv1.DefaultOptionsV1). This implies that v1 is entirely implemented in terms of v2. The implementation of v2 will work hard to ensure bug-for-bug compatibility for every v1 option.

There are several advantages to implementing v1 in terms of v2:

It provides a gradual migration path from v1 to v2.
It reduces the maintenance burden of v1 as there is only one underlying implementation.
It reduces binary bloat as programs linking in both v1 and v2 only use one underlying implementation.
New v2 features that are backwards compatible with v1 (e.g., the omitzero, inline, or format tag options, types that implement jsonv2.MarshalerV2 or jsonv2.UnmarshalerV2, etc.) are immediately available in v1.

In the long-term, we do not plan on ever deprecating the v1 "encoding/json" package. Rather, we will declare the high-level functions and types in that package and point users to the v2 equivalents. Deprecation of v1 functionality would not happen until at least two releases (i.e. 1 year) after v2 is available.

1 reply

dsnet Oct 10, 2023
Collaborator Author

For all future comments about options, continue the discussion here: #63397 (comment)

dsnet · 2023-10-05T17:16:29Z

dsnet
Oct 5, 2023
Collaborator Author

This is further discussion on the behavior of unmarshaling into a non-empty value.

In v2, we aim to provide consistent merge semantics and recommend in UnmarshalerV1 and UnmarshalerV2 that unmarshalers implement some form of merge semantics. Merging as opposed to always clearing is a better default since "always clearing" can trivially be achieved by clearing the top-level Go value being unmarshaled into. On the other hand, merge semantics (if well defined) can be useful for unmarshaling from multiple JSON inputs into the same Go output value.

There are many reasonable semantics for merging, but we should have a consistent approach to how inputs are merged. The merge semantics in v2 takes inspiration from JSON Merge Patch (RFC 7386). At a high level:

An input JSON null clears the target value (i.e., sets it to the zero Go value)
An input JSON object is merged into a target JSON object where input member values are recursively merged into target member values of the same name.
All other input JSON values replace the target JSON value.

For examples of differences between v1 and v2, see this behavior difference test.

6 replies

dsnet Oct 14, 2023
Collaborator Author

A correspondence with JSON Merge Patch is not always achievable and is also not desirable for JSON null.

Correct. That is why our semantic "takes inspiration from JSON Merge Patch (RFC 7386)". It is not defined as being exactly RFC 7286. We want some type of semantic that is easy to explain and implement towards. The current behavior is somewhat incomprehensible.

gazerro Oct 14, 2023

In the case of JSON Merge Patch, JSON null is given special meaning to make it possible to remove a name/value pair when patching. However, when unmarshalling, allowing JSON null to set a value to its zero value doesn't provide any new capabilities beyond what's already possible.

I find that the behaviors of jsonv1 and jsonv2 regarding JSON null are the least expected.

It would be just as easy to explain and implement if JSON null were only allowed if Go nil can be assigned to the value. If it can't, an error would be returned.

dsnet Oct 16, 2023
Collaborator Author

I'm not exactly following what you're proposing.

Adhering strictly to JSON Merge Patch semantics isn't possible since most Go types do not have presence bits. There is simply no way in many native data structures to indicate that a Go struct field, for example, has been deleted.

Were you instead proposing that null simply be rejected as a valid input in general? Would that also include Go pointers and interfaces where there is an obvious meaning of how a JSON null maps to the Go type?

gazerro Oct 16, 2023

I propose accepting JSON null as valid input only for slices, maps, interfaces, and pointers (i.e., where Go nil can be assigned to the type). For all other Go types, JSON null is considered invalid and unmarshal fails with a SemanticError.

doggedOwl Nov 10, 2023

For all other Go types, JSON null is considered invalid and unmarshal fails with a SemanticError

Whether a JSON document (or value) is valid or not is defined outside of the Go type system. What you are proposing is defining another format practically: GO-JSON but definitly not JSON

dsnet · 2023-10-05T17:16:34Z

dsnet
Oct 5, 2023
Collaborator Author

This is further discussion on the marshaling order of Go map entries.

The proposed v2 behavior is for Go maps to marshal in a non-deterministic order, matching the order (or lack thereof) provided by Go map iteration. Non-deterministic output can be made deterministic by setting the Deterministic option.

In contrast, the v1 behavior is deterministic marshaling of maps.

Non-deterministic marshaling is more performant since maps can be marshaled in a truly streaming manner. Any form of deterministic ordering would require sorting the map, which incurs O(n⋅log(n)) runtime and O(n) memory costs. Performance is important in RPC protocols where the ordering of JSON objects does not matter.

However, non-deterministic ordering is detrimental to any use-case that assumes that the serialized output is stable, such as in tests, in the detection of changed configuration files, or in caching. There are several sources of instability in the JSON grammar:

representation of JSON numbers
representation of JSON strings
whitespace between JSON tokens
ordering of JSON object members

Fortunately, RFC 8785 provides guidance for how JSON numbers and strings are to be formatted and v2 complies with that specification. The whitespace is non-existent by default or at least well-specified under WithIndent and WithIndentPrefix. Given that the other sources of instability have stable behavior, a reasonable case could be made to provide deterministic marshaling of Go maps by default.

It comes down to a tradeoff between performance and convenience, with neither benefit clearly outweighing the other.

Vote on this comment for the default Go map ordering:

👍 for non-deterministic ordering (better performance and the proposed v2 behavior)
👎 for deterministic ordering (more convenient and the v1 behavior)

Voting is no guarantee that the most popular behavior be adopted if compelling arguments for a given approach presents itself.

38 replies

zamicol Oct 10, 2023

@dsnet
For objects, JSON keys are defined to be of type string and cannot be of type integer; your latter example is correct. Thankfully, from an idiomatic perspective, using integer-as-string for JSON keys is likely bad practice. While technically possible, it should be avoided.

The correct canonicalization is:

{"0":null,"300":null,"500":null,"70":null,"9":null,"90":null}

dsnet Oct 10, 2023
Collaborator Author

@zamicol, may you kindly stop bringing RFC 8785 into the discussion? Thank you. This thread is about the ordering of Go maps under json.Deterministic. The only relevance of RFC 8785 is for inspiration to the latter, but criticisms about RFC 8785 or the implementation is out of scope here (please start a new discussion as asked for above).

Let's start with arrays. The JSON RFC states that arrays may have mixed types.

I think "mixed types" is being confused for "mixed ordering" or "unordered".
Deterministic ordering must not affect the ordering of JSON arrays, which are "an ordered sequence of zero or more values."

Now let's examine deduplication: deduplication isn't a concern for arrays, but it is for objects. Escapes complicate matters.

Escape sequences are a red herring.

In the case of map[int]string, there is a potential use of ordering based on int. There's no escaping here. (BTW, I'm not saying it's a good idea to sort on numerical ordering or that use of map[int]string is even a good idea. Use of it already exists in v1, and I'm simply raising the question of whether we should sort it specially or not.)

In the case on non-numeric keys, sorting is based on Unicode codepoints of the resulting string (regardless of whether there are any escapes). This can be done based on UTF-8 or UTF-16 (though prior discussion seems to favor UTF-8).

zamicol Oct 10, 2023

arrays, which are "an ordered sequence of zero or more values."

Totally slipped my mind! That's correct.

stop bringing RFC

I'm not understanding, please accept my apology. I will move all discussion regarding the Canonicalize method to a new discussion.

dsnet Oct 10, 2023
Collaborator Author

Thanks @zamicol! For those who want to discuss canonicalization further, see #63397 (comment).

veqryn Oct 12, 2023

As long as we keep the deterministic ordering as an option (I like the contents of my json logs to be ordered alphabetically, after the time, level, and msg), I don't care what the default is and am good with the more performant non-deterministic ordering as default.

dsnet · 2023-10-05T17:16:37Z

dsnet
Oct 5, 2023
Collaborator Author

This is further discussion on how to marshal nil slices and maps.

It is clear from #27589 and #37711 that users need the ability to control whether nil Go slices and maps marshal as either JSON null or empty JSON arrays or objects. However, what nil Go slices and maps should marshal as by default is less clear. The proposed default in v2 is to marshal nil Go slices and maps as empty JSON arrays and objects, while providing a format:emitnull option to opt into the alternative behavior on a per-field basis. We provide FormatNilSliceAsNull and FormatNilMapAsNull options to alter the behavior across the entire Marshal call. Regardless of the default, there will be an option to opt-into the alternate behavior.

Benefit:

Avoids leaking quirks of the Go type system. JSON is a language-agnostic data interchange format. The fact that maps and slices are nil-able in Go is a semantic detail of the Go language. We should avoid leaking such details to the JSON representation. When JSON implementations leak language-specific details, it complicates transition to/from languages with different type systems.

Detriment:

Cannot round-trip marshal/unmarshal slices or maps. Nil slices or maps marshal as empty JSON arrays and objects, which are subsequently unmarshaled as empty (but non-nil) slices. This means that the input and output values are not equal according to reflect.DeepEqual. However, exact equivalence of a marshal-to-unmarshal roundtrip is already not guaranteed in a number of situations such as marshaling Go struct values with non-zero unexported fields. Non-equivalence also means that in the case of maps, the input nil map would result in the allocation of a non-nil map in the output. However, unmarshal is already allocation heavy where this additional allocation may not be noticeable.

Vote on this comment for what the default nil Go slice or map representation should be:

👍 for nil marshaling as an empty JSON object or array (the proposed v2 behavior)
👎 for nil marshaling as JSON null (the v1 behavior)

Voting is no guarantee that the most popular behavior be adopted if compelling arguments for a given approach presents itself.

14 replies

ToadKing Oct 6, 2023

Also, if you're marshalling a field whose type is an interface then you're kind of already saying that this field can be literally any type to begin with, so trying to adhere to a schema for that field seems kinda silly.

efd6 Oct 6, 2023

It's also possible to just make the value be what you want to say.

doggedOwl Oct 6, 2023

the v2 default behaviour is more frendly to consumers. Json gets consumed most of time by clients in other languages and mostly they expect an empty array especially most consumers in JS or python world where working directly on lists is a common idiom and they expect things like arrayItem.forEach to work by default or at least only having to check for the presence or not of the property (ommited or not) and not also it's nullness.

smyrman Feb 5, 2024

Retracted: Would be more suitable to discuss on the #63397 (comment) topic. Also omitzero struct tag appears to handle this.

There are a few cases where at least the difference between an empty array and omitted really matters. E.g. consider the following two MongoDB queries:

{"id": {"$in": []} : Matches no items.
{"id": {}}: Matches all items.

So even if encoding a nil array as an empty array by default, I do not think that an empty array should be omitted by default:

type Comarison struct{
    In []any `json:"$in,omitempty"` // Should only treat `nil` as omitted.
}

rowanseymour Sep 13, 2024

I was initially in favor of this because it seems logical for slices but I'm not convinced it makes sense for maps. Something which is a map on the go side of an API exchange could be a nullable struct on the other side, and sending {} is going to break something.

I think we should definitely make it easier to opt in to marshaling nil slices as empty JSON arrays. It's both tedious and inefficient to force people to turn nil slices into empty slices prior to marshaling. But not convinced that should be default behavior.

dsnet · 2023-10-05T17:16:41Z

dsnet
Oct 5, 2023
Collaborator Author

This is further discussion on how to omit fields during marshal.

Being able to control which fields are omitted is a reasonable feature. However, the challenge is providing the most flexible API for deciding when to omit a field.

Broadly speaking, there are two dimensions that omission can be determined by:

It could be based on the Go type system, where certain Go values are defined as being omitted (e.g., zeros, nil, empty slices, etc.) or certain types that self-declare to be omitted (e.g., an IsZero or ShouldOmit method).
It could be based on the JSON type system, where Go field values that encode as certain JSON values are omitted (e.g., null, "", {}, etc.). Customized representation (e.g., via MarshalJSON) would affect whether the JSON value is to be omitted.

Which approach is the best? Both have legitimate usages and neither covers all of the common use-cases by itself. For that reason, v2 proposes support for both omitzero and omitempty, where the former operates in terms of the Go type system, and the latter in terms of the JSON type system.

Omission with `omitzero`

The omitzero option omits the field if it is the Go zero value or possesses an IsZero() bool method that reports true. Properties of this approach:

The “Go zero value” is well-defined in the Go language. This is in contrast with the v1 definition of omitempty, which is narrowly defined as "false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string". Notably missing from the v1 definition are the zero value representation of structs and arrays.
The “Go zero value” allows us to omit the zero value of Go structs and arrays (#11939). Some types have well-defined zero values (e.g., netip.Addr).
The “Go zero value” allows us to distinguish between a nil versus empty (but non-nil) Go slices and maps, allowing us to omit the former, but not the latter (#22480).
Omitting the "Go zero value" has the nice property that marshaling a Go struct will emit only the non-zero fields such that unmarshaling the JSON output back into a Go struct will generally return the same result (per reflect.DeepEqual).
As an implementation detail, the reflect.Value.IsZero method provides first-class and optimized support for the “Go zero value”. Alternative definitions that exclude unexported fields are both more complicated to explain and not as performant since we would have to implement our own recursive walk through a Go value using reflection. Attempts to exclude unexported fields or non-JSON serializable fields seem to actually want omission defined in terms of the JSON value (see below) and not the Go value.
Respecting an IsZero method allows types to provide their own custom definition of zero-ness. This works well with time.Time.IsZero which handles cases of zero time beyond just being time.Time{}.

Omission with `omitempty`

The omitempty option omits the field if the value would have been encoded as an empty JSON value, which we define as being a JSON null, "", {}, or []. Properties of this approach:

The application of this is recursively effective, allowing omitempty to handle cases that omitzero cannot. For example, consider the following:

type Parent struct {
    Child Child `json:",omitempty"`
}
type Child struct {
    XXX string         `json:"-"`
    Foo string         `json:",omitzero"`
    Bar []int          `json:",omitempty"`
    Baz json.RawMessge `json:",omitempty"`
}
json.Marshal(Parent{
    Child: Child{
        XXX: "ignored",              // omitted since it is ignored by "-" option
        Foo: "",                     // omitted since it is the zero Go value
        Baz: []int{},                // omitted since it encodes as `[]`
        Raw: jsontext.Value("null"), // omitted since it encodes as `null`
    },                               // omitted since it encodes as `{}`
})                                   // thus, this just outputs `{}`

Omitting JSON [] or {} allows omitempty to omit empty Go slices and maps regardless of nil-ness similar to how the v1 operates today (albeit for different semantic reasons).
Whether a field value is to be omitted is dependent on any custom JSON serialization that would occur (e.g., via MarshalJSON).
We do not support omitting arbitrary JSON values as that would require buffering unbounded amounts of JSON text and making it difficult to provide true streaming support. For our definition, every empty JSON value has a limited width (i.e., at most 4 bytes).
Our definition of an empty JSON value does not include 0 since it is ill-defined whether -0, 0e123456, 0.000000, -0.000e+999 are included. Also, 0 is not what most users think of as being "empty".
As an implementation detail, it might seem slow to encode a JSON value only to discard it. In most cases, the internal implementation can short-circuit the actual encoding if it knows that it would have been an empty JSON value.

Composability

Both omitzero and omitempty are composable. You can specify both if you want, where the effect is the logical OR of the two. That is, if the Go struct field would be omitted under either omitzero or omitempty, then it is omitted. In many cases, the behavior of omitzero and omitempty will be identical. We recommend using omitzero when both have the same effect.

Compatibility

The omitzero option is new in v2 and can be introduced without compatibility problems.

The omitempty option in v2 is redefined relative to v1, and this could cause compatibility issues. Despite the change in semantics, they generally produce the same result in most cases except the following:

Go bools and numeric kinds (e.g., int, float32, etc.) cannot be omitted with omitempty in v2 since the proposed definition of an empty JSON value does not include false or 0 (example).
Go pointers and interfaces may be omitted in v2 but not v1 if the underlying value encodes as an empty JSON value. For example, an empty string stored in an interface or a non-null pointer to an empty struct will operate differently under v2 semantics (example).

In both cases, the v1 semantics can be obtained by using the omitzero option.

Thus, we propose the addition of omitzero to the v1 "json" package before the adoption of a v2 "json" package. This will provide time before the arrival of v2 where the author of Go types can migrate their type declarations to use omitzero instead of omitempty for Go bool and numeric kinds. To be clear, we are not proposing that we alter the behavior of omitempty in v1, but only providing a means for which the author of Go types can make their types compatible with both v1 and a future v2.

3 replies

diamondburned Oct 6, 2023

Would it be helpful to have a way to express the three possible states of omitted, null and T as Go types? This would be especially useful when ingesting fields that convey different information depending on those states.

For example, consider a hypothetical Omittable[T] type. We could then define a simple update type:

type EventUpdate struct {
  // OtherID is the event A's other ID.
  // If omitted, then the field is unchanged.
  // If null, then the field has been reset to null.
  // If ID, then the field's value has been updated.
  OtherID json.Omittable[*ID] `json:"other_id"`
}

Users could then explicitly specify whether they want the OtherID field:

// update event that makes no change to OtherID
EventUpdate{}
// an update event that sets OtherID to something else
EventUpdate{OtherID: json.Present(ptr(123))}
// an update event that resets OtherID to null
EventUpdate{OtherID: json.Present(null)}

or detect if a field is present or not:

var update EventUpdate
json.Unmarshal(b, &update)

if !update.OtherID.IsZero() {
  // Add into a SQL UPDATE statement.
  q = append(q, "other_id = ?, ")
  v = append(v, update.OtherID.Value())
}

Implementing an Omittable[T] type would be relatively easy with the v2 API:

// Ommitable marks that a value of its type is either omitted (not in JSON at all)
// or is present and is of type T. NOTE: the user MUST use this with `,omitzero`!
type Omittable[T] struct {
  value T
  valid bool
}

// IsZero returns true if the field should be omitted.
func (o Omittable[T]) IsZero() bool { return o.valid }

// Value returns the present value. If none, then its zero value is returned.
func (o Omittable[T]) Value() T { return o.value }

// Present returns v wrapped in a new Omittable[T] that will not be omitted.
func Present[T any](v T) Omittable[T] {
  return Omittable[T]{v, true}
}

One downside of this implementation is that the user still has to add ,omitzero manually. If the ShouldOmit() function is implemented (#63397 (comment)), then this implementation could be slightly easier.

I'm also not sure if json.Present is a good name. A better name should probably be used.

The main question is: should this use case be covered by an additional type? Or should users write their own type boilerplate when they need to?

mvdan Oct 6, 2023
Collaborator

I'd like to think about optional/omittable/nullable generic types more broadly across std, like #48702 - there is little here that would be JSON-specific. And if such a generic type were to appear later on, I imagine we could add support for it in encoding/json/v2 in a backwards compatible way.

It's worth noting that #60370 was added, but I'd prefer to not simply follow that example and end up with very similar types across a handful of std packages.

diamondburned Oct 6, 2023

I'd like to think about optional/omittable/nullable generic types more broadly across std

I definitely agree! I think in an ideal world, this would be the best solution.

I only suggested a type in encoding/json/v2 because I wasn't sure how practical it would be to add this into the stdlib, given that the proposal itself is for encoding/json/v2.

ianlancetaylor · 2023-10-05T17:38:46Z

ianlancetaylor
Oct 5, 2023
Maintainer

Thanks for the really excellent detailed proposal.

While servers often use JSON as both input and output, there are a number of programs that generate JSON without reading it, and there are a number that read JSON without generating it. The JSON encoding and decoding functions seem largely distinct.
Have you considered the possibility of separate encoding/json/encode and encoding/json/decode packages? They would perhaps both import from a shared encoding/json package for common information.

4 replies

jub0bs Oct 5, 2023

This remark echoes @bradfitz's regrets about oversharing types between client and server in his design of net/http.

dsnet Oct 5, 2023
Collaborator Author

It's unclear to me the benefit of this.

Dead code elimination (DCE) has gotten much better in recent years such that the cost of importing a package (but not using it) has dropped significantly. Thus, splitting this out doesn't seem to benefit binary bloat. That said, the prototype v2 implementation generates the marshal/unmarshal functions together, so depending on only one would link in the implementation for both (today). But that's an implementation detail; we can modify the code to assist in DCE.

The other benefit of splitting it out is to have a conceptual separation between marshal/unmarshal, but doing some brings a few detriments as well:

As it stands, having both in the "json" package makes the call site unambiguous. It's clear what json.Marshal and json.Unmarshal do. However, encode.Marshal and decode.Unmarshal is less clear unless we go with jsonencode.Marshal and jsondecode.Unmarshal, but that stutters quite a bit.
- We could make it non-idiomatic and rely on the package name as a verb and have it be encode.JSON or decode.JSON, but we'd be making use of a potentially contended namespace for "encode" and "decode". How would we extend this same pattern to XML or YAML? It still bothers me that I can't tell whether rand.Read refers to "math/rand" or "crypto/rand".
json.Marshal is almost the entire opposite of json.Unmarshal. The "json" package provides a unified place to describe that mapping. I'd argue that there's closer correlation between Marshal and Unmarshal than there is between http.Client and http.Server.

mvdan Oct 6, 2023
Collaborator

We could also write a regression test that would build a tiny Go program only using json.Marshal and then check that json.Unmarshal and the other decoding functions and types aren't included in the binary - and vice versa. This would keep us honest and ensure that we don't make any changes in the future that could tie the encoder and decoder implementations together.

mikeschinkel Oct 7, 2023

"It still bothers me that I can't tell whether rand.Read refers to "math/rand" or "crypto/rand".

This. Very much this.

abhinav · 2023-10-05T18:22:10Z

abhinav
Oct 5, 2023

Thanks, @dsnet. I've been looking forward to this discussion!

One minor thing I want to bring up is the naming of the V2 MarshalJSON and UnmarshalJSON methods:

type MarshalerV2 interface {
	MarshalJSONV2(*jsontext.Encoder, Options) error
}

type UnmarshalerV2 interface {
	UnmarshalJSONV2(*jsontext.Decoder, Options) error
}

I'd like to suggest methods without "V2" in their names. Two years down the line, if someone writes a type implementing only the V2 methods, I don't think "V2" should be present in their signatures. That's a small degree of noise that'll exist solely for historical reasons. I understand that being able to implement both V1 and V2 interfaces is a hard requirement so we cannot re-use the name MarshalJSON, but we can probably find something else.

To start the conversation on the interface method names, how about EncodeJSON and DecodeJSON?

type MarshalerV2 interface {
	EncodeJSON(*jsontext.Encoder, Options) error
}

type UnmarshalerV2 interface {
	DecodeJSON(*jsontext.Decoder, Options) error
}

20 replies

dsnet Oct 10, 2023
Collaborator Author

Alternatively, carrying forth @prashantv's idea...

These are the current set of declarations with V1 and V2 suffixes:

DefaultOptionsV2
MarshalerV1
MarshalerV2
MarshalerV2.MarshalJSONV2
UnmarshalerV1
UnmarshalerV2
UnmarshalerV2.UnmarshalJSONV2
MarshalFuncV1
MarshalFuncV2
UnmarshalFuncV1
UnmarshalFuncV2

DefaultOptionsV2 quite literally describes v2 semantics, so I think it's fine and actually more explicit that it carries a v2 suffix. Also, this option would only ever be used when you're dealing with a v1-to-v2 migration. If you're in a pure-v2 world, then you'll never need to reference this.

For the others, they could potentially be renamed as:

MarshalerV1 -> Marshaler
MarshalerV2 -> MarshalerTo
MarshalerV2.MarshalJSONV2 -> MarshalerTo.MarshalJSONTo
UnmarshalerV1 -> Unmarshaler
UnmarshalerV2 -> UnmarshalerFrom
UnmarshalerV2.UnmarshalJSONV2 -> UnmarshalerFrom.UnmarshalJSONFrom
MarshalFuncV1 -> MarshalFunc
MarshalFuncV2 -> MarshalToFunc
UnmarshalFuncV1 -> UnmarshalFunc
UnmarshalFuncV2 -> UnmarshalFromFunc

However, this introduces a discrepancy between the above and the existing set of top-level functions:

Marshal
MarshalWrite
MarshalEncode
Unmarshal
UnmarshalRead
UnmarshalDecode

We can't rename both MarshalWrite and MarshalEncode to MarshalTo,
nor can we rename both UnmarshalRead and UnmarshalDecode to UnmarshalFrom.

To be honest, I personally don't think it's problematic for MarshalerV1, MarshalerV2, UnmarshalerV2, and UnmarshalerV2 to carry the V1 and V2 suffixes since they're interface types and are rarely referenced in user code. I can see why there's a distaste to having V2 in the method name, though, since this appears in all user-code implementing custom JSON.

mikeschinkel Oct 11, 2023

"I personally don't think it's problematic for MarshalerV1, MarshalerV2, UnmarshalerV2, and UnmarshalerV2 to carry the V1 and V2 suffixes since they're interface types and are rarely referenced in user code."

Agreed.

"I can see why there's a distaste to having V2 in the method name, though, since this appears in all user-code implementing custom JSON."

Confirming your perception on this, at least for me.

smyrman Feb 5, 2024

I think it's more important for the names to be clear then for them to order well.

As a curiosity though, If we go for FormatJSON and ParseJSON, then FormatJSON would order before MarshalJSON and ParseJSON would order before UnmarshalJSON. Same goes with Parser v.s. Marshaller and Formatter v.s. Unmarshaller. so the new method would at least order before the "old" methods, if not directly above.

As another alternative, the V1 interface could be implemented either as a private type or as an implicit interface similar to As and Is in the error package. This way old names would not show up in the docs at all, and the docs could just document that v1 interfaces are detected without publicly redeclaring them.

iamdlfl Feb 5, 2024

I like @prashantv suggestion, but could we re-order the words so it is MarshalToJSON and UnmarshalFromJSON? I know the ordering would be a bit off, but I feel like it would be more clear. As a new developer I had to regularly think about which method meant what. Also MarshalJSONTo makes it sound like it is taking JSON input and vice versa for UnmarshalJSONFrom.

dillon-giacoppo Feb 17, 2024

To have a type keep its V1 interface whilst also enable forward compatibility with V2 (assuming naming collisions), there could be a bridging interface:

type UnmarshalerV2Supplier interface {
    AsUnmarshalerV2() UnmarshalerV2
}

An existing V1 type such as:

type Foo int

func (a *Foo) UnmarshalJSON(val []byte) error { ... }

Can be extended like so:

type foo2 Foo

func (a *Foo) AsUnmarshalerV2() json.UnmarshalerV2 {
    return (*foo2)(a)
}

func (a *foo2) UnmarshalJSON(dec *jsontext.Decoder, opts json.Options) error {...}

And a type that does not require backwards compatibility of course needs only implement v2 interface directly:

type Bar int

func (a *Bar) UnmarshalJSON(dec *jsontext.Decoder, opts json.Options) error {...}

Could also have a UnmarshalerV1Supplier interface to do the same principle in reverse. That seems less ideal for
backwards compatibility with older Go versions as UnmarshalerV1Supplier would not be understood by the existing json package.

This avoids the suffix and having to invent new names for each new major version. As V1 code is upgraded to the V2
interfaces, it can drop the *Supplier. The ugliness is only retained whilst both versions need to be concurrently supported, i.e. initial migration period.

Not suggesting this is necessarily a good solution, but shows there is viability in overloaded names without losing compatibility.

jub0bs · 2023-10-05T19:01:34Z

jub0bs
Oct 5, 2023

@dsnet Are you planning to use some form of the pattern known as "functional options" for configuring Encoders and Decoders? If so, I have some thoughts:

AFAIK, the pattern is not yet used in the standard library. And, as much as I find the pattern useful in the right situation, it remains controversial in the community. Is there a risk of alienating the pattern's detractors?
Calling functional options on the hot path of the program can introduce performance issues. Granted, in most programs that encode/decode JSON, performance bottlenecks will lie elsewhere, but if one objective is to improve over v1's performance, this consideration should factor in the design.
NewDecoder and NewEncoder may never fail at this early stage of the design, but options added later may change that. In order to future-proof the API, I suggest designing them as fallible by making them also return an error:
```
func NewDecoder(io.Reader, ...Options) (*Decoder, error)
func NewEncoder(io.Reader, ...Options) (*Encoder, error)
```
Since multiple options can be specified in those functions' variadic parameter, shouldn't the type's name be singular (Option rather than Options)?
If some options only make sense for decoders, others only for encoders, and yet others for both, multiple options types may make sense.

More of my thoughts about how to make most of the functional-options pattern are available online in video format. Interesting that we agree on declaring the option type as an opaque interface and that we both (ab)use type aliases! 😉

42 replies

dsnet Oct 25, 2023
Collaborator Author

At this point. I'm going to somewhat disengage from the conversation. Others are free to continue discussing.
I work on Go as an external contributor and don't have the bandwidth to continually engage in long threads of response.
Unlike most other golang discussions, which are hosted by members of the Go team, I'm not employed by Google.
Thank you for your time.

mikeschinkel Oct 26, 2023

@dsnet — Once again, thank you for taking the time to engage.

Given additional debate would be moot, I accept your position on a hybrid API.

However, please allow developers to build their own Hybrid API; that will only take a tiny bit of code change.

Here is the (updated) PR to that effect, ready to merge:

go-json-experiment/json#17

Since it is an additive approach as you acknowledge, allowing developers to implement their own will allow the Go team to see if there is strong interest in this, or if nobody other than a few like me care enough to write their own hybrid API.

But if developers cannot do it themselves, it will be hard to ever know.

-Mike

P.S. I would not even see the need to document this extensibility. People who want it will find it and use it. And if enough people do want to use it, then we can document it.

willfaught Oct 29, 2023

Marshaler, Unmarshaler, Encoder, and Decoder structs (not separate option types, the actual types) that contain option fields don't have this issue.

To be clear, you would actually need both a Marshaler and EncodeMarshaler type, where Marshaler has just options related to marshaling and EncodeMarshaler and options related to both marshaling and encoding.

@dsnet I don't follow. The encoder option fields would be mixed in with the marshal option fields in Marshaler, then copied to the Marshaler's Encoder option fields. You only need to deal with one option struct when directly un/marshaling or de/encoding.

Which is why each type has its own fields. If there's some obvious overlap, then fine, make a shared struct, but not at the expense of useless fields in some contexts, as pointed out here.

There is overlap in a number of options. A shared type makes the ergonomics of calling worse. Anything that introduces more references to declarations in general hurts ergonomics.

So then don't do that. Have each type have its own fields.

That sounds less clean and more verbose to me.

The variadic approach is objectively still less verbose:
json.Marshal(v, json.Deterministic(true))
versus
json.Marshaler{Deterministic: true}.Marshal(v)

In that context, if I remember correctly, I was addressing the complexity and verbosity of the API declarations, not call sites.

For those specific call sites, sure, but in practice, you'd never write that. You'd write:

m := json.Marshaler{Deterministic: true}
m.Marshal(v)
...
m.Marshal(v)
...
m.Marshal(v)

Using functional options, you'd have to write:

json.Marshal(v, json.Deterministic(true))
...
json.Marshal(v, json.Deterministic(true))
...
json.Marshal(v, json.Deterministic(true))

Your counterargument might be that you'd instead write:

o := json.Deterministic(true)
json.Marshal(v, o)
...
json.Marshal(v, o)
...
json.Marshal(v, o)

But in that case, it's a wash in terms of verbosity.

However, methods hung off an options struct has an additional problem in that you can't as easily use them in an expression.

I would argue that it's idiomatic to construct a value and then mutate it before use, as in option fields. We can find plenty of examples in the stdlib. Not everything needs to be expression-friendly. I imagine that most uses of options will be to get v1 behavior, and if I recall correctly, that requires more than one option to be changed, so I would expect most instances to be created in this way.

Speaking of idioms, are there any examples of functional options in the stdlib that you can point to? I'm not aware of any.

v2 options should be declared with v2. If v2 has options to behave like v1, then that's fine, that's the design, and v2 should reflect that.

If you don't want v2 to be "polluted" by v1, then don't make it behave like v1. This is trying to have your cake and eat it too, and as I've pointed out in my top-level thread, which you haven't addressed here yet, there are downsides to doing that (as there always are with trying to have your cake and eat it too).

This is just going to be a point that we're going to have to disagree on.

I don't think so. If you can think of a concrete rebuttal, I'm all ears.

Option structs [...] interfere with dead-code analysis.

Can you explain this?

If the *Marshalers type is a field in MarshalOptions, it must be linked in if MarshalOptions is referenced. With the variadic API, it's possible to implement it in such a way that caller-specified marshalers is entirely dead-code eliminated if WithMarshalers is never called.

OK, this sounds like it's about reflect, and not wanting to link that in, unless needed.

concrete binary and memory space costs of importing reflect

Importing reflect cannot be quantified as an X number of bytes of binary bloat because it is not simply a fixed number of instructions that needs to be linked in.

Thanks for the great explanation. It makes sense why we'd want to minimize links to reflect in general. So if jsonv2 needs reflect just to do its basic job, then what option contains a direct reference to reflect that you're trying to avoid if possible? I assume it's a jsontext option? Skimming above, I don't see mention of reflect anywhere in encoding options, and none of the encoding options seem to do reflection-related things.

This sounds more and more like a framework with plugins.

Whether or not it uses dependency injection is purely an implementation detail. The "json" API does not expose this detail. Thus, it is not a "framework".

Well, it does expose the detail in the option itself, which is part of the total API, but I agree that the core API does not. This was a minor point, and I can see if you squint at it right, it's not a framework. Fair enough. I actually like dependency injection.

jsonv1 required reflect, so it's unsurprising that jsonv2 would too

Yes, and that has been constraining on a few use cases for embedded or mobile applications. A flaw of v1 is not an argument for why it should exist in v2.

It might be if we insist on v2 being able to emulate v1 behavior. There are other considerations, like API ergonomics and complexity.

Saving 50KB of RAM while running in iOS with a 150MB RAM limit is not a valid reason, in my humble opinion

It's funny you say that...

I've read that, which is what I was vaguely alluding to when I wrote that. I couldn't find the link at the time, so thanks for sharing. Like I said above:

I'm looking for a concrete scenario, like "we have to simulate the universe in 32 bytes of RAM because of X, Y, and Z." Saving 50KB of RAM while running in iOS with a 150MB RAM limit is not a valid reason, in my humble opinion. If you need to do crazy things or need extreme optimization, don't expect the stdlib to cater to all your needs. Or don't expect backward compatibility with jsonv1.

We should either go with the stdlib catering to all needs, or go with backward compatibility with jsonv1, but not both, because going with both has this complicated API design. Users can just convert their JSON from v1 to v2 instead, and use v1 in the meantime.

I don't follow. Can you explain how slog is currently planned to work with jsonv2?

See #63397 (reply in thread)

This is just sticking a jsonopts.Options field in slog. You could do the same thing with a Marshaler.

If some other package has its own configuration of jsonv2 that it wants users to override

That doesn't work in packages (e.g., "slog") where it manages the marshaling of JSON on your behalf.

You can just pass it a Marshaler. Configure it however you want. If you want to "override" some custom behavior that's done by default, read the documentation to know what that is, then configure the Marshaler appropriately.

type SomeSlogStructSomewhere struct {
    ...
    // Marshaler is...
    // If unset, these options are used:
    // - ...
    // - ...
    // - ...
    Marshaler *json.Marshaler
    ...
}

That's all you need to tailor a Marshaler to exactly what you need for slog.

It's not a loss of type safety, it's a loss of immutability.

Immutability is a sub-category of type safety.

There are languages that have immutable values and no types or type system, so this isn't true on its face. I would say that immutability is a property that is a consequence of the absence of semantics that enable mutation. Type systems can describe values and expressions in such systems, but they aren't the same thing.

This subject will probably go off-topic, so I'd be happy to continue this rabbit hole elsewhere if you'd like. :)

I don't see how. Can you explain?

In an earlier version of the v2 "json" prototype, we exposed an option called:
EscapeRune func(rune) bool
that specifies which runes to escape in a JSON string. (we eventually removed this overly powerful option and replaced it with
EscapeForJS and EscapeForHTML). For performance, we pre-executed the function on all ASCII characters and built a small bit-map of which ASCII characters needed escaping, which vastly improved performance as it avoided a function call for a majority of characters. However, if the EscapeRune option were mutable (or non-deterministic), then the cache would become incorrect.

I don't see how mutability of boolean options relates to this, though. As you said, EscapeRune is gone, precisely because it prevented this optimization. And I don't see how EscapeRune pertains to the option struct vs. function options decision. You can specify EscapeRune either way, so EscapeRune as a functional option would have the same mutability problem.

Since jsontext.Encoder is passed to json.MarshalerV2.MarshalJSONV2, it would be problematic if the implementation of MarshalJSONV2 can change the behavior of encoding by toggling jsontext.Encoder.Indent. It makes no sense to change such parameters partway through encoding.

See "So Don't Do That" above.

I disagree.

Looking back, I misread your original comment. I see what you mean now about other types' implementations of MarshalJSONV2 mutating Encoder. That is indeed a potential problem, one that I don't have an easy answer for right now. Perhaps an Encoder interface used in that context would prevent mutation of the fields. I'm not clear on why Options is required for MarshalJSONV2, but assuming they are, then there's the question of how to supply the option struct too.

Good point!

mvdan Oct 30, 2023
Collaborator

Note that v1 compatibility has already been discussed elsewhere, like #63397 (comment). I worry that this thread has a big scope and is getting very long, to the point that we're duplicating a number of the other discussions.

willfaught Oct 30, 2023

@mvdan Unless you can be specific, I don't see an issue here. Functional options touch on compatibility, optimization, and other issues.

Let's not micromanage the partial overlap of comments. Topics naturally evolve and touch on other topics, or refer back to themselves. If you're not getting any more value out of this discussion, then I suggest unsubscribing. This thread, and this discussion in general, have wound down naturally without intervention. I don't think @dsnet plans to post more, and I don't have anything else to add.

josharian · 2023-10-05T19:09:35Z

josharian
Oct 5, 2023
Collaborator

If we're thinking to the next decade, I'd like to make sure it will be easy to add support for HuJSON/JWCC.

(I know Go isn't a trend-setter, but I'd even love for it to be available from the beginning.)

17 replies

dsnet Oct 11, 2023
Collaborator Author

Perhaps, I'm not understanding how you plan on extending PeekKind in a hypothetical future. Right now, it only ever returns 9 possible characters:

'n' for null
'f' for false
't' for true
'"' for string
'0' for number
'[' for starting array
']' for closing array
'{' for starting object
'}' for closing object

It would be an unfortunate hindrance to usages today if all callers of PeekKind need to care about colons and commas. One of the nice properties of JSON over JWCC is that commas and can always be perfectly inferred.

creachadair Oct 11, 2023

I probably don't have any deep insights here. My general experience has been that many tools using JWCC only care about it as input, and the "obvious" ways to efficiently parse and represent it do not adapt well when you want to write it back out (especially if you also need the ability to edit). I think/agree there is real value in distinguishing the "one way" case from the "two way" case in the API.

Extending the scanner and parser to (efficiently) handle the JWCC extensions is straightforward, but you need quite a bit of API mass to preserve a comment-decorated AST, and if you want tools to be able to add, edit, and remove comments, the memory footprint of that representation is substantial. (One could almost certainly do better than I did in that package, but the problem is not specific to it).

Moreover, even basic JSON API functions like marshal/unmarshal don't map to JWCC very well: For example, there is no nice way to "compactify" JWCC text, without mangling the legibility or throwing away the comments. Any useful round-tripping of JWCC is probably going to need nice indentation, and then we get into the whole "jwcc fmt" issue where the API either has to be opinionated (and invite an unending future of option requests) or add a gallimaufry of knobs.

My intuition is that none of this belongs in a (baseline) JSON package, and that despite its superficial similarity and one-way compatibility with JSON, JWCC as an objet d'arte should be treated as its own thing, and programs that care to round-trip it should use a different API.

nigeltao Oct 12, 2023
Collaborator

One of the nice properties of JSON over JWCC is that commas and [colons??] can always be perfectly inferred.

Perhaps we are misunderstanding each other, but with JWCC, commas and colons can also always be perfectly inferred. Provided that, when writing JWCC, you're writing in some canonical comma style. One option for "canonical comma style" is to simply output commas as per JSON. Another option is to always write commas after an element. Either way, you're writing per an opinionated formatter, the same way that gofmt writes an opinionated format even though, as input, Go's semi-colons are optional. Just like gofmt, it doesn't really matter what the canonical style is, just that there is one.

If you're worried that round-tripping will not produce byte-for-byte identical output, then that's already a problem for JSON (due to whitespace, which your package jsontext discards; see also capital-C RFC 8785 Canonicalization), not just JWCC.

nigeltao Oct 12, 2023
Collaborator

Right now, it only ever returns 9 possible characters:

Well, 10, if you include returning const invalidKind Kind = 0 at EOF. Or maybe more if you want to discriminate EOF from other io.Reader errors. (See also the "negative Kind values mean different errors" idea elsewhere in this issue discussion.)

how you plan on extending PeekKind in a hypothetical future

That nitpick aside, my proposal is to add ' ' (the U+0020 SPACE byte) as a Kind for filler. Code like this:

for d.PeekKind() != '}' {
  // etc
}

becomes

for {
  if k := d.PeekKind(); k == '}' {
    break
  } else if k == ' ' {
    d.SkipFiller()  // or repurpose the existing SkipValue method,
                    // or allow a Value to represent filler.
    continue
  }
  // etc
}

But if you're calling Decoder.ReadToken in a loop, instead of rolling your own Decoder.PeekKind based loop, then you're unaffected. Likewise if you're using the higher level marshal/unmarshal API.

In case my point wasn't clear, I don't think we can extend PeekKind in the future. If we want to have the ability to support comments, whether from day zero or in the future, we'll need to have filler tokens from day zero.

nigeltao Oct 12, 2023
Collaborator

Having said all that, "many tools using JWCC only care about it as input" sounds plausible to me. Perhaps a "90% of the benefits, 10% of the costs" middle ground would have a boolean Options to allow comments and trailing commas, only on the decode side, and simply treat them the same as whitespace (and not preserved on round-trip). No need then for a filler Kind.

That would let people opt in to consuming JWCC. Producing JWCC (including automated edits) would be punted to a separate package.

josharian · 2023-10-05T19:15:04Z

josharian
Oct 5, 2023
Collaborator

What do you think about having an inverted control API as well (i.e. instead of writing json, you get a reader)?

Relevant historical discussion: https://gophers.slack.com/archives/C0VP8EF3R/p1655155603882039, #51092 (comment). There's a general sense that the Right Fix is something general purpose involving io.Pipe, but error handling remains a unsolved issue.

I'm bringing it up again here because generating JSON for use as an HTTP POST body is so common.

2 replies

rogpeppe Oct 5, 2023
Collaborator

I wonder whether the coroutine proposal might end up helping in that respect.

dsnet Oct 5, 2023
Collaborator Author

When I saw the coroutine package, I immediately started thinking about how it could solve the inverted io.Reader and io.Writer problem, but I haven't quite come to terms with exactly how. Since this problem exists outside of just "json", but also with gzip, base64, etc., it'd be great if we had a single solution for all packages.

josharian · 2023-10-05T19:19:13Z

josharian
Oct 5, 2023
Collaborator

Dealing with union-typed JSON (e.g. a given field is either a string or a map[string]string or a map[string]SomeStruct, but nothing else) is a real pain point when working with APIs designed by people using loosely-typed languages. Might there be some first class support for union types?

27 replies

mvdan Oct 8, 2023
Collaborator

Also, a thought about the proposal process and release schedule: would it be a dealbreaker to, for example, ship this encoding/json/v2 API as-is without first-class support for unions or sum types? As far as I can tell, it should be possible to add support later, as an incremental step. For example, we could later start treating some structs as unions like @josharian's example with a new option like json:",union". If that sort of later addition doesn't seem possible for any reason, please say so.

I only mention this because there's a balance at play here; if we try to cram too many features into encoding/json/v2 from day one, the proposal process and implementation will take longer, and it might take an extra year for any of this to release. Holding up the proposal seems right if there are any blockers or design issues that will become a limiting factor later on, though.

szabba Oct 8, 2023

There's also multiple ways people encode union types ( { "type": "variant", "data": { ... } }, { "variant": { ... } }, [ "variant", { ... } ], { "type": "variant", ... }) and the possibility of some form of them landing in Go. Holding back on adding this functionality seems wise.

dsnet Oct 10, 2023
Collaborator Author

As far as I can tell, it should be possible to add support later, as an incremental step. For example, we could later start treating some structs as unions like @josharian's example with a new option like json:",union". If that sort of later addition doesn't seem possible for any reason, please say so.

I'm not sure we would even need an explicit union option.

Suppose a future Go adds interface{ A, B } where Go reflection allows us to see that this is a list of types, then we can have "json" unmarshal different JSON kinds into the appropriate type. This assumes that the "json" package can discern that JSON numbers go to type A and JSON strings go to type B. This is possible for types without any custom JSON methods. Once you add custom representation, it gets tricky. We would need some way to annotate the interface type to know the mapping. As an aside, this is one reason why it would be useful to have type tags similar to struct field tags.

Since type lists in interface types isn't possible today, there are zero occurrences of them in the wild. So long as we add support in "json" for unions the moment they land in the language, then we're okay.

dsnet Oct 11, 2023
Collaborator Author

The conversation up to now has focused mostly on the Go language representation. I'm going to pivot and take it from the perspective of a JSON parser.

Dynamic JSON roughly falls into two categories:

A dynamic JSON value where the resulting concrete Go type is determined entirely based on JSON kind itself (e.g., use of a Go string for a JSON string and a []string for a JSON array such as spec.StringOrArray).
- The important thing to note is that we know which Go type to use simply by peeking at the kind of the upcoming token. This is a relatively easy case where we could potentially provide first-class support in the "json" package. That said, we'll probably want to see where native union types in Go eventually ends up.
A dynamic JSON value where the resulting concrete Go type is determined by some other JSON value. This situation is harder and breaks down into two sub-categories: whether we can know for sure the type in a forward parsing (easy) and whether we can't (hard):
- A. In the easy case, we are guaranteed to encounter the type name before we encounter the dynamic value. For example, one such construction is something like {"TypeName": ...} where TypeName determines the concrete Go type to use and ... is unmarshaled into that concrete Go type. The syntactic construction guarantees that TypeName must be encountered first. Thus, this can always be handled in a purely streaming manner.
- B. In the hard case, we may not see the type name before we encounter the dynamic value. This is unfortunately the more common case since many dynamic JSON uses constructions like {"type": "TypeName", "value": "..."} (e.g., google.protobuf.Any). Since JSON objects are unordered name-value pairs, there is no guarantee that we will see the "type" member before the "value" member. It is entirely valid that we encounter {"value": ..., "type": "TypeName"}, where we don't know the concrete Go type to use by the time we encounter the value itself. This greatly complicates the implementation where it must implement a double parse to temporarily hold the value in a jsontext.Value until it encounters the type (which also implies O(n²) for recursive situations). As an optimization, we can hope that most JSON inputs have the "type" come first, but that doesn't change the fact that we still need the double-parse fallback.
- This form of dynamic JSON value is highly varied that it's unclear whether the "json" package should provide first class support for this. For example, not every schema uses a "type" field (e.g., protobuf uses "@type") or whether there is a "value" field or whether the value is inlined at the same level as the "type". The v2 package does at least make this situation easier to handle manually, though.

ydnar Oct 11, 2023

I was pleased to learn you can implement streaming unmarshaling of union types with UnmarshalFuncV2 by creating an unmarshal func where T is a pointer to an interface:

func unmarshalAttr(dec *jsontext.Decoder, v *Attr, opts json.Options) error {
	var proxy struct {
		Width  *int    `json:"width"`
		Height *int    `json:"height"`
		Name   *string `json:"name"`
	}
	err := json.UnmarshalDecode(dec, &proxy)
	if err != nil {
		return err
	}
	switch {
	case proxy.Width != nil:
		*v = Width(*proxy.Width)
	case proxy.Height != nil:
		*v = Height(*proxy.Height)
	case proxy.Name != nil:
		*v = Name(*proxy.Name)
	}
	return nil
}

type Attr interface {
	isAttr()
}

Full working code here: https://github.com/ydnar/scratch/blob/main/cmd/json-union/main.go

soypat · 2023-10-06T01:40:59Z

soypat
Oct 6, 2023

Is it possible to specify a max stack/object depth and reject the JSON on hitting the limit? I don't see an Option that provides this functionality

1 reply

dsnet Oct 6, 2023
Collaborator Author

Probably not in the initial release, unless we get around to implementing it.

We did set aside the possibility of adding jsontext.WithByteLimit and jsontext.WithDepthLimit options. Also, the proposal for at least something like jsontext.WithByteLimit has been accepted for v1.

ToadKing · 2023-10-06T06:23:16Z

ToadKing
Oct 6, 2023

UnmarshalRead consuming the entire input is a good change, but will there be a way exposed in the json package for getting the old behavior? Or will you have to roll your own using jsontext? Multiple JSON objects in a single stream are uncommon but maybe not uncommon enough to expose some way for users to do it explicitly. Perhaps as a jsonopts?

1 reply

dsnet Oct 6, 2023
Collaborator Author

The equivalent behavior in v2 would be:

json.UnmarshalDecode(jsontext.NewDecoder(r), &v)

where you create a one-off jsontext.Decoder and only unmarshal the next JSON value in the stream and ignore the rest.

It's a little bit more typing than:

json.NewDecoder(r).Decode(&v)

but I'd imagine that this is the far less common use-case.

mitar · 2023-10-06T07:15:36Z

mitar
Oct 6, 2023

Thank you for a very detailed proposal. I read through it but maybe I missed it, but it seems to me that current proposal does not provide access to struct tags for MarshalJSONV2 and UnmarshalJSONV2? To me that was always the biggest limitation. That if I wanted to customize a bit how JSON was marshaled or unmarshaled (e.g., do some custom validation on values), all users of my struct would not be able to provide struct tags to fine control that custom implementation.

I see that there is new format struct tag, which seems like something which will be passed to few standard structs, but I missed somewhere in Options where you access that format? Am I missing something?

Or, in other words, I think Options should have a way to access json struct tag (or maybe even all struct tags, so that one can use additional struct tags, e.g., if json is missing but yaml exists, use yaml) associated with the value currently being marshaled or unmarshaled.

7 replies

dsnet Oct 6, 2023
Collaborator Author

In turns out we had a more in-depth discussion about recursive format flags on 2021-04-09. Fundamentally, it's complicated and we weren't happy with what we could come up with at the time.

nemith Oct 6, 2023

This was a problem that we wanted to solve when dealing with generated structs in Thrift (specifically FBThrift).

We had a number of endpoints that expected the numerical (value) representation of enum (implemented with const iotas) and a number of them that wanted a the enum/const name.

example:

type MyEnum int

const (
   MyEnum_VARIANT1 = iota
   MyEnum_VARIANT2 = 1
   MyEnum_VARIANT3 = 2
)

Some times when encoding this i want the value to be a string of "VARIANT1" and sometimes i would want the value to be a numerical 1 (or even a string of "1"). Without duplicating every type and every structure that refers to that type

There was no seemingly easily way to influence how these deeply nested structures would end up emitting these values when encoding with json since the format was tied to the type and couldn't be influenced at runtime/call site.

It would be nice to be able to not modify a structure to influence the format options in some way. Not all structures are going to be in your control or flat enough to easily just embed and extend.

dsnet Oct 6, 2023
Collaborator Author

@nemith We haven't implemented it in v2 "json" yet, but #56235 would solve the enum situation. For example, see #56235 (comment).

nemith Oct 6, 2023

@dsnet I wasn't aware of that proposal and it seems very interesting but doesn't change the fact that I cannot choose what format to encode at runtime. Duplicating generated structs and embed "enum" types is still needed.

How it seems that decoding seems like it could be more permissive.

Thrift solved this by completely re-implemented the json encoding (which isn't bad cause it already knows about types, etc). Protobufs also has it's own JSON encoding so perhaps this isn't a big deal.

rogpeppe Oct 9, 2023
Collaborator

@dsnet I wasn't aware of that proposal and it seems very interesting but doesn't change the fact that I cannot choose what format to encode at runtime. Duplicating generated structs and embed "enum" types is still needed.

Actually, if I understand your problem correctly, this is possible with the current proposal, because it's possible to define custom marshalers and unmarshalers for a given type at runtime that aren't directly associated with that type.

For example: https://go.dev/play/p/mY3vJOPAtPG

maxtreaming · 2024-04-08T12:25:10Z

maxtreaming
Apr 8, 2024

Current implementation of json Unmarshall requires to create a variable that will be used to unmarshall data to.

Since go has generics, we could have that step wrapped into a helper function:

func UnmarshallTo[T any](data []byte) (T, error) {
	var result T
	err := json.Unmarshal(data, &result)
	return result, err
}

8 replies

maxtreaming Apr 8, 2024

@Kiura thanks for pointing me this, it's the same :)

mikeschinkel Apr 8, 2024

@maxtreaming, @Kiura — Sounds like a really nice idea, and an easy win IMO.

It would be an improved API for when you want to use a more functional approach with no side effects, and reusing memory is not a major concern, e.g.:

HandleJSON(UnmarshallTo[Foo](`{"text":"Hello world"}`))

Note that my example changes the param from []byte to an interface that allows string|[]byte, since we have Generics, after all. 🤷‍♂️

Kiura Apr 8, 2024

Using string|[]byte is a really nice idea. Reduces the boilerplate code when you already have a string, not []byte or as in the example you used above.

maxtreaming Apr 9, 2024

I was also thinking about some constrains on the generic type, so that the compilation could already reject unsupported ones.

dsnet Apr 23, 2024
Collaborator Author

This is being discussed in #59053, so let's move further discussion there.

Of relevance for v2, we would need 3 additional functions (for consistency):

func UnmarshalTo[T any]([]byte) (T, error)
func UnmarshalReadTo[T any](io.Reader) (T, error)
func UnmarshalDecodeTo[T any](*jsontext.Decoder) (T, error)

glenjamin · 2024-05-03T09:45:14Z

glenjamin
May 3, 2024

Thanks for this proposal, the level of detail and thoughtfulness is impressive.

I've read over everything and I couldn't see any discussion of this point, so I'll raise it here. I suspect the answer will be "out of scope for initial release", but since everything else is in here I figure it's worth noting.

As far as I can tell there are 3 options for how JSON names map to struct names:

The new default, strict 1-1 mapping of struct name to JSON key
The old default, opt in, semi-fuzzy case-insensitive mapping
Respect the name given in the json tag

In most systems/ecosystems, there is a standard convention for naming of keys, often snake_case.

I believe it would be useful to be able to configure Marshal or Unmarshal with a strict mapping that uses user-specified functions to perform the translation.

This would allow the caller of marhshal/unmarshal to enforce the naming convention without requiring a tag on every struct field.

4 replies

mvdan May 3, 2024
Collaborator

This was briefly brought up earlier in #63397 (comment), although GitHub loves hiding old comments once a thread reaches a certain size, so it's not surprising that it's buried at this point unless you find the button to load older comments.

The problem with a func with arbitrary logic is that it likely prevents the use of caching for efficiency. Both the encoder and decoder do as much work ahead of time as possible, including preparing indexes so that decoding JSON object key-values can be quickly mapped to a Go struct field. Otherwise for every incoming JSON object key we'd need to do a Go struct field linear search, resulting in quadratic behavior - plus the func call.

One potential solution could be to constrain this feature to only map from Go field name to JSON key name in a deterministic way, so that the json package could call the function on all struct field names ahead of time, and still do its caching the same way. For example, replacing the struct field tags in struct {FooBar int `json:"foo_bar"`; FooBaz int `json:"foo_baz"`} with something like TransformFieldNames(func(name string) string { return toSnakeCase(name) }).

One could say "just use code generation or an IDE plugin to insert the struct field tags", but that's a rather awkward solution. It's not like stringer, which can generate a separate file.

Or, alternatively, options which directly implement common transformations like "snake case" or "lower case". I think the best middle ground would be to support the func flexibility, but also provide transformation funcs that implement common practices like snake case.

I still think we can easily add this to json/v2 after the initial proposal; the existing proposal is large enough as is, and it's already taking significant effort to actually get it merged. But it shouldn't be a problem to add on top of the current design as a separate proposal, and I'd probably be in favor.

glenjamin May 3, 2024

This was briefly brought up earlier in #63397 (comment),

Ah interesting, I did see that thread but skipped over it as it was talking about replacing tags entirely, rather than merely reducing the need for them

One potential solution could be to constrain this feature to only map from Go field name to JSON key name in a deterministic way, so that the json package could call the function on all struct field names ahead of time, and still do its caching the same way. For example, replacing the struct field tags in struct {FooBar int json:"foo_bar"; FooBaz int json:"foo_baz"} with something like TransformFieldNames(func(name string) string { return toSnakeCase(name) }).

Yeah, this is how I'd want it to work, presumably that would mean that the reflection cache key would still need to include a reference to the transformer function?

I still think we can easily add this to json/v2 after the initial proposal; the existing proposal is large enough as is, and it's already taking significant effort to actually get it merged. But it shouldn't be a problem to add on top of the current design as a separate proposal, and I'd probably be in favor.

👍

mvdan May 3, 2024
Collaborator

Yeah, this is how I'd want it to work, presumably that would mean that the reflection cache key would still need to include a reference to the transformer function?

Yes, in some way. Right now the cache is only keyed by Go type, so we would have to teach the caching logic to separate cache entries per transform func. Although it could get tricky if we want to be efficient with different funcs that return the same results for a particular type.

myaaaaaaaaa May 16, 2024

Another option would be to let the user handle the caching problem:

type Customer struct {
	Name  string
	Email string
}

var jsonCustomer = json.MustCompile[Customer](json.RenameField(strings.ToLower))

func main() {
	...
	jsonCustomer.Marshal(customer)
	...
}

deefdragon · 2024-06-10T01:11:52Z

deefdragon
Jun 10, 2024

I've encountered a timestamp output from an external API that I've been unable to resolve with the current format tags. The timestamp is sent as a json number, not a string. Would it be possible to have time.Time values with format:unix,unixnano,etc. tags be able to parse from a string and from a number?

5 replies

smyrman Jun 10, 2024

EDIT: Retracted. This comment was written without reading the question properly. Please ignore the content.

If you read the issue description you will find:

The time.Time type accepts a "format" value which may either be a Go identifier for one of the format constants (e.g., "RFC3339") or the format string itself to use with time.Time.Format or time.Parse (#21990). It can also be "unix", "unixmilli", "unixmicro", or "unixnano" to be represented as a decimal number reporting the number of seconds (or milliseconds, etc.) since the Unix epoch.

As a side note, the common way to solve this with the current JSON package, is to wrap the time.Time type to implement your own Unmarshal logic:

type UnixTime struct{
    time.Time
}

func (t *UnixTime) UnmarshalJSON(data []byte) error {
   // parse integer / float value.
   // if it fails, fallback to:
   return json.Unmarshal(data, &t.Time)
}

E.g. an approach you could follow if you use separate data structures for JSON parsing and the rest of your application.

dsnet Jun 10, 2024
Collaborator Author

If you want to format a Unix timestamp as a JSON string containing a JSON number, you can do:

struct {
    Time time.Time `json:",string,format:unix"`
}

You mentioned wanting to unmarshal from either a JSON number or a JSON string. The v2 implementation today allows for it with the string tag option, but we'll be reverting that behavior to only allow for JSON strings if the string option is specified. This was an oversight in the v2 implementation as it was a unnecessary divergence from v1 behavior.

deefdragon Jun 13, 2024

@smyrman, I've been working with the experiment library for some time using the string representation of JSON timestamps and different formatting strings, both encoding and decoding. That works quite well. The issue I've just encountered tho was when it was a JSON number being returned, and I don't have control over the API.

I probably should have been clearer when describing my problem however and included the actual data, so for fullness allow me to be specific.

I have a JSON object being returned from the API roughly as follows after the request

{"time":17181795451234}

and a go struct roughly as follows that I need to get the data into.

type Resp struct {
	Time *time.Time `json:"time,omitempty,format:unixmilli"`
}

Currently, attempting to decode the given JSON into the struct using the experiment library resulted in an error.

The ask was that it be possible to have the library decode a timestamp automatically into the format specified if its a number {"time":17181795451234} OR a string {"time":"17181795451234"}. (that is, being able to automatically accept either without having to specify which.) IMO the default flexibility gained would be a win, but I'm not sure if there are performance/other implications that would negate that.

That said @dsnet, Regarding the string tag option. Are you saying that it will ONLY decode into a string if that option is specified? Passing a number wont decode into the timestamp? Or are you saying that including the string tag will cause it to parse numbers and strings as I've asked for here?

dsnet Jun 13, 2024
Collaborator Author

Using string will only accept JSON string and reject a JSON number.

var v1 struct { V time.Time `json:"V,string,format:unix"` }
fmt.Println(json.Unmarshal([]byte(`{"V":"1234"}`), &v1)) // no error
fmt.Println(json.Unmarshal([]byte(`{"V":1234}`), &v1))   // error
var v2 struct { V time.Time `json:"V,format:unix"` }
fmt.Println(json.Unmarshal([]byte(`{"V":"1234"}`), &v2)) // error
fmt.Println(json.Unmarshal([]byte(`{"V":1234}`), &v2))   // no error

As mentioned, this is currently not the behavior of HEAD for github.com/go-json-experiment/json, but we will be changing the behavior to what is shown above.

The reason for this change is because v1 acts the same way today.

var v1 struct { V int `json:"V,string"` }
fmt.Println(json.Unmarshal([]byte(`{"V":"1234"}`), &v1)) // no error
fmt.Println(json.Unmarshal([]byte(`{"V":1234}`), &v1))   // error
var v2 struct {	V int `json:"V"` }
fmt.Println(json.Unmarshal([]byte(`{"V":"1234"}`), &v2)) // error
fmt.Println(json.Unmarshal([]byte(`{"V":1234}`), &v2))   // no error

smyrman Jun 13, 2024

@deefdragon, your initial question was good and too the point; I managed to miss it. Please consider my previous comment as retracted.

pappgyorgy · 2024-06-11T11:21:13Z

pappgyorgy
Jun 11, 2024

I have a similar question like (#63397 (comment)) regarding limiting the size of the JSON. What do you think about limiting the number of leaf elements in a JSON and reject it if the limit is reached? Did you consider this possibility already? If not, then what is the opinion on adding this possibility to the new release?

4 replies

dsnet Jun 11, 2024
Collaborator Author

It's conceivable that we add a WithLengthLimit option, but this is the first time I've heard of a desire to limit based on the number of leaf elements. I'm somewhat more hesitant about WithLengthLimit for performance reasons. By the nature of the limit. this has to check on almost every JSON token.

On the other hand, WithByteLimit mainly only needs to be checked when fetching data from the underling io.Reader, and WithDepthLimit mainly only needs to be checked when encountering a ArrayStart or ObjectStart token. Thus, the impact of those options on performance is lower.

mitar Jun 11, 2024

WithByteLimit can be done with io.LimitReader?

dsnet Jun 11, 2024
Collaborator Author

That's a good question, take a look at #56733.
Per the proposal:

Typically, the advice to avoid reading excessively large values into memory when passing an untrusted io.Reader is to wrap the reader in a io.LimitedReader. For encoding/json.NewDecoder this is not necessarily a reasonable approach, since the Decoder may be intended to read from a long lived stream (i.e. a net.Conn) where the user may not want to limit the total amount of bytes read from the Reader across multiple reads, but does want to limit the number of bytes read during a single call to Decoder.Decode

mitar Jun 11, 2024

Thanks for the reference. Very insightful.

bjg2 · 2024-06-18T09:59:19Z

bjg2
Jun 18, 2024

Not sure if this was mentioned or not (it was not as far as I could see but thread is too big): there should be an option of adding mapping function for struct field name -> JSON name and vice versa.

Simple example is that go forces your field to be capitalized to be unmarshalled (as that is needed for it to be public), while usually JSONs do not follow that pattern (they are either lowerCammelCase or snake_case, almost never UpperCammelCase). You can override name in the json tag, but you end up writing tag for every single field in your big codebase, and then writing devops that checks tags and generates code as well... And you end up with more complex situation which is much harder to maintain.

5 replies

mibk Jun 18, 2024

That’s an unfortunate characteristic of Go symbol semantics. One thing I’ve learned to appreciate over the last decade is the ability to grep the symbol name for its usage. However, with the existence of a concept such as a mapping function, the ability to grep is lost.

dsnet Jun 18, 2024
Collaborator Author

This has been brought up before in #63397 (comment).

One of the major challenges is that accepting an arbitrary function to rename JSON fields interferes with the ability to pre-compute information about types since functions in Go are incomparable, and even if they were, we have no guarantees that they are deterministic.

bjg2 Jun 19, 2024

Not sure I understand the challenge completely, as I am not sure why would potential 'non-deterministic' mapping function be an issue? I mean, I get why it is the issue with the programmer that wrote non-deterministic function, but I'm not sure why would that be an issue with the system itself, there is a plethora of ways to use golang in wrong, undesired way, and against the documentation. For example: I had an issue when I was using rand.Rand's Intn() concurrently, which is documented to be a problem, and I would never say golang has a bug for it (example #51015). There is a number of other cases, I mean, I don't see a problem in adding a quality of life improvement that does not have to be used and that has to be used as mentioned in documentation when used.

I think that solution similar to this should be considered, though it is not pure, as golang has its own specific issue connected to the fact that capitalized field means public. To be clear, I love that capitalized field means public, it is very readable, but we cannot ignore the issues it brings.

Because alternative is just terrible. In our current codebase we have 2979 json tags, almost all of them only mapping UpperCammelCase to lowerCammelCase. There is maybe even more text in json tags than in struct fields themselves, that cannot be by design.

glenjamin Jun 19, 2024

My understanding is this:

Currently every struct has only 1 JSON encoding. This allows the generated mapping to be cached with a pretty straightforward key.

If the mappings are customizable, then any such customization needs to be taken into account when building the cache, which is not a trivial problem.

Solutions do exist, and I believe adding this in the future hasn't been completely ruled out, but the complexity is enough that it's unlikely to be in scope for the initial release.

There's also some discussion about this here: #63397 (comment)

bjg2 Jun 19, 2024

I understand. Maybe the 'good enough' and 'easy enough' quality of life improvement would be not to have mapping function but only few predetermined default mappings, 'UpperCammelCase' (as it is at the moment), 'lowerCammelCase' (only lowercases first letter), and 'snake_case' (lowercases each uppercased letter + adding _ before each non-first uppercased letter). That might be called hacky, but I guess it would solve 99% of the problems and avoid the mentioned issues...

Sorry to everyone because writing under this thread, but there is a number of threads pointing to the other threads and I just cannot choose the right one so I'm continuing with this one, as it is currently alive. :)

FeldrinH · 2024-07-09T13:35:01Z

FeldrinH
Jul 9, 2024

Is there any hope that encoding/json/v2 could include the ability to mark fields as required? One issue that I consistently run into is that I want to enforce that a field must be explicitly declared in the JSON, even if it contains a zero value (the purpose of this is usually detecting accidental misuse of an API or changes to the structure of data returned from a downstream API). I can kind of achieve this right now by using a pointer field and checking if it is nil, but this is tedious and error prone. Also I find using pointers for values that should actually never be nil kind of dirty.

What I would really like to have is something like this:

type struct Data {
    Field int `json:",required"`
}

And then unmarshalling a JSON object that does not have this field (e.g. {}) would produce an error. But unmarshalling {"Field": 0} would not.

10 replies

doggedOwl Jul 9, 2024

I want to enforce that a field must be explicitly declared in the JSON, even if it contains a zero value (the purpose of this is usually detecting accidental misuse of an API or changes to the structure of data returned from a downstream API).

Both of these things belong elsewhere in the chain and not on the consumer. You also state that the encoding/json already does validation but it only does type validation not content validation.
Opening the door to content validation means opening it to flooding with request that "required" means "not zero" or that it means also in a range or it means that a required string is an email etc etc.

FeldrinH Jul 9, 2024

I think the canon way to do this is to use a pointer field (playground)

As I said in the initial post, this does work (as long as you don't need to distinguish between explicit null and a missing field) but I find this dirty because now I have made a field nillable for no other purpose than to check that it is not nil. To a reader looking at the type and not the validation logic this could give the misleading impression that the value is actually optional and Data{Field: nil} is a valid value.

FeldrinH Jul 9, 2024

Both of these things belong elsewhere in the chain and not on the consumer.

Where do you think they belong? In my opinion, when I am writing an API, I want to make it as hard to misuse as possible and part of that is validating that the consumer has supplied all the required fields. When I am consuming a downstream API I might not have control over when and how the output of that API changes (especially if it is a poorly documented external API). In those cases explicitly validating that they have not removed or renamed a field that I use is useful, because it can detect some unexpected API changes early and alert me to the need to update my code.

You also state that the encoding/json already does validation but it only does type validation not content validation. Opening the door to content validation means opening it to flooding with request that "required" means "not zero" or that it means also in a range or it means that a required string is an email etc etc.

I think validating the existence of a required field is kind of a special case that should be handled by the unmarshalling logic, because the unmarshalling logic is the last place where field presence is explicitly known. After that any validation has to depend on various assumptions about the unmarshalling logic, such as assuming that a missing field in the JSON results in a zero value in the struct.

willfaught Jul 9, 2024

As was said, it seems like you can already accomplish what you want with pointer types and a struct validation tool. For example:

type T1 struct {
    F *string `json:"f,omitempty" validate:"required"`
}

Regardless of how dirty it feels, a pointer type is correct and appropriate because the value might not be there. If you want to avoid handling pointers elsewhere in your code, then copy the values to a non-pointer equivalent:

type T2 struct {
    F string
}

var t2 T2 = T2{F: *T1(t1).F}

andoks Jul 10, 2024

As I said in the initial post, this does work (as long as you don't need to distinguish between explicit null and a missing field) but I find this dirty because now I have made a field nillable for no other purpose than to check that it is not nil. To a reader looking at the type and not the validation logic this could give the misleading impression that the value is actually optional and Data{Field: nil} is a valid value.

Indeed, you did point this out in the OP, sorry.

The trouble is that if I have a struct like
type struct Data {
    Field int
}
then after unmarshalling, if Field == 0, it is impossible to tell if the field was missing or explicitly set to 0. No external validation can solve this problem.

I agree, in particular because from what I can deduce, one has to change how one decodes all sibling properties too, which, if this is the only way to do this without an external library (or writing the from JSON type very unergonomically), may be an argument for adding such a feature as required. I would expect the required tag to work for both pointer types and non-pointer types, where if a pointer-type is tagged with required, it can be explicitly set to null (but the "key" MUST be present regardless), while if a non-pointer type is tagged as required, the json must be set to a valid value of the expected type.

Show code - writing the from JSON type very unergonomically

https://go.dev/play/p/DOAWLt38b_o

package main

import (
	"encoding/json"
	"fmt"
	"math"
	"strings"
)

type DataJSON map[string]interface{}

type Data struct {
	Field int
}

func MakeValid(d DataJSON) (Data, error) {
	if v, found := d["Field"]; !found {
		return Data{}, fmt.Errorf("JSON was not valid Data, missing field \"Field\"")
	} else if v == nil {
		return Data{}, nil
	} else if vv, ok := v.(float64); !ok {
		return Data{}, fmt.Errorf("failed decoding field %q, field was of type %T, not int", "Field", v)
	} else if math.Trunc(vv) != vv {
		return Data{}, fmt.Errorf("failed decoding field %q, value was not an integer = %v", "Field", vv)
	} else {
		return Data{Field: int(vv)}, nil
	}
}

func main() {
	ins := []string{"{}", "{ \"Field\": null }", "{ \"Field\": 42 }", "{ \"Field\": 13.37 }"}

	for _, in := range ins {
		dec := json.NewDecoder(strings.NewReader(in))

		var out DataJSON
		err := dec.Decode(&out)

		if err != nil {
			panic(err)
		}

		if p, err := MakeValid(out); err != nil {
			fmt.Printf("%q was not valid Data = %#v - error was %v\n", in, out, err)
		} else {
			fmt.Printf("%q was valid Data = %#v\n", in, p)
		}
	}
}

timbray · 2024-07-09T16:58:40Z

timbray
Jul 9, 2024

On Jul 9, 2024 at 9:44:42 AM, Juhan Oskar Hennoste ***@***.***> wrote: I think validating the existence of a required field is kind of a special case that should be handled by the unmarshalling logic, because the unmarshalling logic is the last place where field presence is explicitly known. After that any validation has to depend on various assumptions about the unmarshalling logic, such as assuming that a missing field in the JSON results in a zero value in the struct

I find this argument very convincing. Requiring the presence of a field for successful unmarshaling feels like such a basic, obvious thing. Message ID: ***@***.***>

…

0 replies

hiendaovinh · 2024-08-08T04:33:46Z

hiendaovinh
Aug 8, 2024

I'm looking for a way to define a context-aware options which change the behavior of custom MarshalJSONV2 based on the context.Context. The usecases are mostly omitting/adding fields based on the context of serialization.

0 replies

stevenh · 2024-08-15T22:26:50Z

stevenh
Aug 15, 2024

Has there been any discussion about context support to enable a caller to interrupt a potentially long running encode / decode?

2 replies

andoks Aug 16, 2024

Is not such cancellation considered the data source's responsibility? E.g
https://go.dev/play/p/22x-bSsdV1Q

playground code

// You can edit this code!
// Click here and start typing.
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"time"
)

func main() {
	ctx, stop := context.WithTimeout(context.Background(), 1500*time.Millisecond)
	defer stop()
	in := NewCancellableReader(ctx, 100*time.Millisecond)
	dec := json.NewDecoder(in)
	for {
		var obj struct {
			Message string `json:"msg"`
		}
		err := dec.Decode(&obj)
		if err != nil {
			fmt.Printf("decoding element from stream of object failed, exiting loop, error was: %v\n", err)
			break
		}
		fmt.Printf("Received object: %+v\n", obj)
	}
}

var _ io.Reader = (*CancellableReader)(nil)

type CancellableReader struct {
	ctx          context.Context
	numBytesRead uint64
	interval     time.Duration
}

func NewCancellableReader(ctx context.Context, interval time.Duration) *CancellableReader {
	return &CancellableReader{
		ctx:          ctx,
		numBytesRead: 0,
		interval:     interval,
	}
}

var ErrorContextDone = fmt.Errorf("context was done")

func (cr *CancellableReader) Read(b []byte) (int, error) {
	t := []byte("{\"msg\": \"Hello World!\"}")

	select {
	case <-cr.ctx.Done():
		return 0, ErrorContextDone
	case <-time.After(cr.interval):
		start := cr.numBytesRead % uint64(len(t))
		end := len(t)
		numBytesRead := copy(b, t[start:end])
		cr.numBytesRead += uint64(numBytesRead)
		return numBytesRead, nil

	}
}

stevenh Aug 16, 2024

Nice, no need for it then :)

doggedOwl · 2024-08-16T07:55:44Z

doggedOwl
Aug 16, 2024

Is this discussion going anywhere? The discussion here seems that has leveled down and I hope almost 1 year is more than enough to make it to proposal phase.

3 replies

andoks Aug 16, 2024

It might be a bit in a limbo, quoting the original proposer:

At this point. I'm going to somewhat disengage from the conversation. Others are free to continue discussing.
I work on Go as an external contributor and don't have the bandwidth to continually engage in long threads of response.
Unlike most other golang discussions, which are hosted by members of the Go team, I'm not employed by Google.
Thank you for your time.

From here: #63397 (reply in thread)

It might have been better if this issue was locked with a last comment explaining the status/hiatus (if no one took over as the sheperd for this issue / proposal, I have not seen anyone who have done that)

mvdan Aug 19, 2024
Collaborator

I believe Joe was just replying to that thread there, not to the entire discussion.

All of us working on the v2 proposal and implementation work on it in our spare time, so sometimes that means progress is paused while we are busy. You can see recent activity over at https://github.com/go-json-experiment/json.

Some work remains before we are ready to move onto a proposal. The main chunk of it is to implement encoding/json v1 behavior in the new code so that, once v2 is approved and merged, we only maintain one json package with two public APIs rather than two parallel json packages - given that we must continue maintaining v1 practically forever for backwards compatibility.

andoks Aug 20, 2024

Apologies, it seems like I misunderstood the context of the response. Great to hear it is still ongoing, and thank you for all your hard work 👍

jdemeyer · 2024-10-14T13:17:58Z

jdemeyer
Oct 14, 2024

Concerning time.Duration: isn't the proposed default format "1h2m3.456s" rather Go-specific? Given that JSON is regularly used to communicate to non-Go software, that's a strange default.

A more logical default would be the number of seconds (you didn't mention it, but I'm assuming a floating-point number of seconds).

12 replies

dsnet Nov 13, 2024
Collaborator Author

I've called out in the past that the problem with ISO 8601 is that it's not a single grammar, but a family of different grammars with various optional grammars that are up to communicating parties to agree upon which is valid or not. As one can imagine, this is not good for interoperability since there is no singular definition of what a duration means. This is why RFC 3339 exists (see section 5.5), which is to restrict ISO 8601 down to a single date-time grammar that the Internet can agree on.

The JavaScript flavor of ISO 8601 is specified here:
https://tc39.es/proposal-temporal/#sec-temporal-iso8601grammar

Given that there is a paragraph that says, "the grammar deviates from the standard given in ISO 8601 in the following ways ..." further underscores my point that ISO 8601 is not a strict grammar. This deviation is fine (and perhaps arguably necessary) in order to strictly define a grammar for good interoperability.

I haven't studied the grammar in-depth yet to understand how it is similar to or different from other ISO 8601 flavors in common use.

dsnet Nov 13, 2024
Collaborator Author

Unfortunately, TC39 does not define the length of units like "years", "months", "weeks", etc., so we're still left with a problem in Go where we might need to only support a subset of the grammar up to days.

deefdragon Nov 14, 2024

If I'm remembering correctly (proposal has gone on for a while), the marshaler/unmarshaler allows both format tags for customization, as well as the ability to override/use custom stuff for specific types right? That should be usable in this case to allow for different duration formats & implementations of year/month/week etc. based on user need.

I think the only question is what should be the default, which should probably be the json1 values for comparability reasons?

andoks Nov 14, 2024

Unfortunately, TC39 does not define the length of units like "years", "months", "weeks", etc., so we're still left with a problem in Go where we might need to only support a subset of the grammar up to days.

It is indeed unfortunate, but in my (limited) experience, parsers of durations do not require you to convert a unit to the larger one (seconds to minutes, minutes to hours etc) when duration increases to that point, so I do not think only outputting in days, hours, minutes, seconds and fraction of seconds should be a challenge (for most) in practice

Maybe even following postels law could make sense here: "be conservative in what you send, be liberal in what you accept", meaning parsing weeks etc etc following whatever is the determined to be the best or most used interpretation, but always outputting only in days/hours/minutes/seconds/fraction-seconds.

I think the only question is what should be the default, which should probably be the json1 values for comparability reasons?

At least that is a solid argument. I would also argue though, that it should be iso by default (to encourage more adoption), and a compatibility tag could be added so that when adopting v2, developers must change the tag appropriately for the fields that must be v1 compatible, while the rest becomes iso. But that I feel is more more of question for the maintainers to decide on, with regards to weighing backwards compatibility vs encouraging standard formats by default.

I am fully satisified with a standard struct tag way of specifying that the duration is to be represented as iso duration both on write and on read so that a) I don't need to use an external library with wrapper types just for iso duration formatting b) I am more sure of that the iso-formatted durations are formatted correctly

dsnet Nov 19, 2024
Collaborator Author

parsers of durations do not require you to convert a unit to the larger one (seconds to minutes, minutes to hours etc) when duration increases to that point, so I do not think only outputting in days, hours, minutes, seconds and fraction of seconds should be a challenge (for most) in practice

It matters for Go, since we fundamentally need to store a syntactic duration in ISO 8601 into a Go time.Duration. Therefore, we need to assign semantic meaning to a "year", "month", and "day". We could arbitrarily reject time durations that use years, months, and days, but that's an odd stance to hold.

timbray · 2024-10-15T03:47:59Z

timbray
Oct 15, 2024

Urgh. I would be shocked if golang serialized timestamps and durations in a way that wasn't RFC3339 (and thus ISO8601) conformant.

…

On Mon, Oct 14, 2024, 6:23 AM Mitar ***@***.***> wrote: rfc3339 has a very similar duration format, so not sure if it is really Go specific? I have also seen it elsewhere in various HTTP protocols. — Reply to this email directly, view it on GitHub <#63397 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEJE6QRN26GT6EQBGY4ZLZ3PAUDAVCNFSM6AAAAAA5US42KSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJTGY4TKOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

jason-gigastar · 2024-11-05T23:33:10Z

jason-gigastar
Nov 5, 2024

My impression is that one of the goals of this package is to better handle the common 3 variable state such as:

Not passed (field not in JSON)
Passed as null value (field = "null")
Passed as non-null value (field = "alpha")

These states can be elegantly modeled by a double-pointer but it seems like that is not supported as detailed below.

Aside on double-pointer:

Some may find a ** gross but having gone down the path of a 'generic' Null type and getting it to sort-of-work I found the path to be an akward waste of time in so many ways. I feel like a double-pointer works for all types in Go AFAIK more elegantly and with performance benefits (but not my primary concern). If packages expect double pointers it can reduce integration efforts and boilerplate adapter/conversion. I accept that pointers have to be checked but Go seems to love that with 'if err != nil' everywhere and the universal nature of pointers seems more appealing than all the possible Null types.

To demonstrate this concept, I extended the omit fields example with relevant excerpts below. The jist of it is that this package's 'omitzero' tag allows the conversion to JSON string to behave as I'd hoped - specifically that a field with a zero-value double pointer is omitted from the output.

Extended example:

	ptrStringNil := (*string)(nil)
	alpha := "alpha"
	ptrStringVal := &alpha
	// Demonstrate behavior of "omitzero".
	b, err := json.Marshal(struct {
		// ...
		DblPtrNotExist  **string `json:",omitzero"`
		DblPtrExistNull **string `json:",omitzero"`
		DblPtrExistVal  **string `json:",omitzero"`
		// ...
	}{
		// ...
		DblPtrNotExist:  nil,           // omitted=true,  Like PATCH with missing field
		DblPtrExistNull: &ptrStringNil, // omitted=false, Like PATCH with field value: null
		DblPtrExistVal:  &ptrStringVal, // omitted=false, Like PATCH with field value: "alpha"
	})
	err = (*jsontext.Value)(&b).Indent("", "\t") // indent for readability
	require.NoError(t, err)
	const expected = `{
	"Struct": {},
	"Slice": [],
	"Map": {},
	"Pointer": "",
	"Interface": null,
	"DblPtrExistNull": null,
	"DblPtrExistVal": "alpha"
}`
	require.Equal(t, expected, string(b))

HOWEVER, In the following I found that the conversion from JSON string to struct does not behave as I'd like for the case of a *nil (aka null value) as the "null" string value results in a zero-value double pointer whereas I'd hoped for a *nil.

Please let me know if I'm missing something here.

Unit test verifying behavior ("inner nil" case fails from JSON to struct):

func Test_Json_2way(t *testing.T) {
	alpha := "alpha"
	fieldVal := &alpha
	var fieldNil *string

	// This works as expected, the field is omitted in the json string
	verifyExamplePatch(t, "outer nil", true, &ExamplePatch{Field: nil})

	// This works as expected since all values are set
	verifyExamplePatch(t, "inner non-null", true, &ExamplePatch{Field: &fieldVal})

	// The interesting case:
	// * v1 allows 3 field states (exists, null, non-null) but 'json.Marshal' ignores the concept of exists as modeled
	// by an outer/double ptr and writes the value as "null" in the same way as an inner/single ptr. It would
	// be better if a nil outer ptr caused the field to be omitted in the json string. The 'json.Marshal' behavior
	// loses state (both !exists and null yield the same result) and the receiver always sees the field as null or non-null
	// * v2 solves the case where v1 fails via 'omitzero' but when reading from string the inner ptr is ignored
	// and only the outer ptr is set to nil
	verifyExamplePatch(t, "inner nil", true, &ExamplePatch{Field: &fieldNil})
}

func verifyExamplePatch(t *testing.T, scenario string, expectPass bool, input *ExamplePatch) {
	// Serialize type to string
	str, err := ToJson2String(input)
	require.NoError(t, err, scenario, input)
	require.NotEmpty(t, str, str, scenario, input)

	// Deserialize string to type
	out, err := FromJson2String[ExamplePatch](str)
	require.NoError(t, err, str, scenario, input)

	require.Equal(t, expectPass, input.Equal(out), scenario)
}

type ExamplePatch struct {
	Field **string `json:"field,omitzero"` // 3 states of (exists, null, non-null)
}

func (a *ExamplePatch) Equal(b *ExamplePatch) bool {
	if (a.Field == nil) != (b.Field == nil) {
		return false // 1 of 2 ptrs is nil
	}
	if a.Field == nil && b.Field == nil {
		return true // Both outer ptrs nil
	}
	if (*a.Field == nil) != (*b.Field == nil) {
		return false // 1 of 2 inner ptrs is nil
	}
	if *a.Field == nil && *b.Field == nil {
		return true // Both inner ptrs nil
	}
	return **a.Field == **b.Field // Compare inner values
}

func ToJson2String[T any](input T) (string, error) {
	jsonBytes, err := json.Marshal(input)
	if err != nil {
		return "", err
	}
	return string(jsonBytes), nil
}

func FromJson2String[T any](input string) (*T, error) {
	var item T
	err := json.Unmarshal([]byte(input), &item)
	if err != nil {
		return nil, err
	}
	return &item, nil
}

Update: If I combine this package and a generic Null type 🙁 that allows the 3 states to flow through struct -> JSON -> struct.

6 replies

earthboundkid Nov 7, 2024

You can just do any(n.V).(isZeroer) instead of using the reflect.Value.Interface.

func (n Maybe[T]) IsZero() bool {
	if n.Present {
		return false
	}
	if z, ok := any(n.V).(isZeroer); ok {
		return z.IsZero()
	}
	rv := reflect.ValueOf(&n.V)
	return rv.Elem().IsZero()
}

rogpeppe Nov 7, 2024
Collaborator

Indeed - as my TODO comment mentions, there are potentially lots of ways of making it more efficient.

jason-gigastar Nov 7, 2024

Thanks for the suggestion. I've had a generic Null type for awhile but it has awkward aspects that I was hoping to simplify away.

It seems double-pointer is unlikely to be a near term solution since AFAIK Go's type assertions lack a generic distinction between pointer and non-pointer types. This leads to a fall-back to either a generic Null type, reflection, or an explicit list of types with pros/cons to each but all being less good IMO.

It seems like Go has aspirations to continue improving the type assertions, I hope they do. I'm relatively new to Go and it has many positives but when it comes to generics I think of what's possible with C++ using templates and meta-programming and I feel my power tools are missing a power plug at times.

dsnet Nov 12, 2024
Collaborator Author

My impression is that one of the goals of this package is to better handle the common 3 variable state

I wouldn't say that is a goal of the package, but the omitzero feature (which as of today will be supported in v1 as well) does set the groundwork to make this better supported. It's not clear to me yet that the json package is responsible for making support for this better today, or whether there should be a Go-wide optional type for this.

Also, the variability in use-cases is wide such that it's also not clear what the "right" solution. For example, at tailscale we only needed 2 of the states:

Not passed (field not in JSON)
Passed as non-null value (field = "alpha")

So we created https://pkg.go.dev/tailscale.com/types/opt#Value. It would be fairly easy to create a variation on the opt.Value type that handles the 3 states.

mgnsk Nov 13, 2024

I've used this kind of semantics for forms.

Required - the field must exist in JSON.
RequiredNullable - the field must exist but may be explicitly null.
OptionalNonNullable - the field may or may not exist but must not be null.

It's not strong enough to represent all states at the type system level but was sufficient to use reflection to extract rules for a JSON schema validator that gets run before unmarshaling into the struct. By reusing the omitempty tag, it is also compatible with marshaling back to JSON.

// All fields that don't use the omitempty tag are required. Pointers and pointer types are
// nullable and required. Optional fields must be pointers or pointer types noted with the
// omitempty tag in which case they they are not nullable. The meaning of null in this
// context is a missing field.

type Data struct {
	Required            bool  `json:"a"`
	RequiredNullable    *bool `json:"b"`
	OptionalNonNullable *bool `json:"c,omitempty"`
}

timbray · 2024-11-14T02:49:01Z

timbray
Nov 14, 2024

On Nov 13, 2024 at 2:36:46 PM, Joe Tsai ***@***.***> wrote: Unfortunately, TC39 does not define the length of units like "years", "months", "weeks", etc., so we're still left with a problem in Go where we might need to only support a subset of the grammar up to days.

This only applies to durations, right? -T Message ID: ***@***.***>

…

3 replies

dsnet Nov 14, 2024
Collaborator Author

Correct. The Go implementation of JSON has always had support for RFC 3339 for timestamps.

cespare Nov 14, 2024
Collaborator

@timbray I'd encourage you to reply to these discussion threads using GitHub. Each of your emails creates a new, separate top-level thread in this discussion (and that may be why the other ones haven't gotten any responses).

Honestly I don't understand why GitHub even lets email replies to discussions work if they're this broken.

timbray Nov 14, 2024

Urgh sorry will do. I even knew that.

Gunni · 2024-11-18T15:35:50Z

Gunni
Nov 18, 2024

Hey, I wanted to mention an issue i had a lot when working with json.

When unmarshalling I want to know if I received all the keys, or be able to mark keys as mandatory, and consider it an error if some keys are missing when unmarshalling. Zero values are not ok randomly.

I mean:

Value received, not zero value = fine
Value received, value is zero value = equally fine
Key not part of json = error <== this

Does this make sense?

1 reply

dsnet Nov 18, 2024
Collaborator Author

I believe what you're asking for has been proposed as discussed in #19858.

I'm personally not convinced that this is something that needs to happen at deserialization time. The fact that protobuf had this feature and it has been regarded as a mistake seems to suggest that baking it in at the serialization layer is the wrong approach.

It seems that the right approach is to preserve presence in the data structure of the value and for a separate mechanism to do validation. With such an approach, we can building in other forms of validation that people have occasionally asked the JSON layer to try and do.

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

encoding/json/v2 #63397

dsnet Oct 5, 2023 Collaborator

Background

Missing functionality

API deficiencies

Performance limitations

Behavioral flaws

Proposal

Goals

Overview

The "jsontext" package

Overview

Values and Tokens

Encoder and Decoder

Options

Errors

The v2 "json" package

Overview

Default behavior

Struct tag options

Type-specified customization

Caller-specified customization

Options

Errors

Experimental implementation

Stability

Performance

Replies: 87 comments · 504 replies

dsnet Oct 5, 2023 Collaborator Author

dsnet Oct 10, 2023 Collaborator Author

dsnet Oct 5, 2023 Collaborator Author

dsnet Oct 14, 2023 Collaborator Author

dsnet Oct 16, 2023 Collaborator Author

dsnet Oct 5, 2023 Collaborator Author

dsnet Oct 10, 2023 Collaborator Author

dsnet Oct 10, 2023 Collaborator Author

dsnet Oct 5, 2023 Collaborator Author

dsnet Oct 5, 2023 Collaborator Author

Omission with omitzero

Omission with omitempty

Composability

Compatibility

mvdan Oct 6, 2023 Collaborator

ianlancetaylor Oct 5, 2023 Maintainer

dsnet Oct 5, 2023 Collaborator Author

mvdan Oct 6, 2023 Collaborator

dsnet Oct 10, 2023 Collaborator Author

This comment has been hidden.

This comment has been hidden.

dsnet Oct 25, 2023 Collaborator Author

mvdan Oct 30, 2023 Collaborator

josharian Oct 5, 2023 Collaborator

dsnet
Oct 5, 2023
Collaborator

Replies: 87 comments 504 replies

dsnet
Oct 5, 2023
Collaborator Author

dsnet Oct 10, 2023
Collaborator Author

dsnet
Oct 5, 2023
Collaborator Author

dsnet Oct 14, 2023
Collaborator Author

dsnet Oct 16, 2023
Collaborator Author

dsnet
Oct 5, 2023
Collaborator Author

dsnet Oct 10, 2023
Collaborator Author

dsnet Oct 10, 2023
Collaborator Author

dsnet
Oct 5, 2023
Collaborator Author

dsnet
Oct 5, 2023
Collaborator Author

Omission with `omitzero`

Omission with `omitempty`

mvdan Oct 6, 2023
Collaborator

ianlancetaylor
Oct 5, 2023
Maintainer

dsnet Oct 5, 2023
Collaborator Author

mvdan Oct 6, 2023
Collaborator

dsnet Oct 10, 2023
Collaborator Author

dsnet Oct 25, 2023
Collaborator Author

mvdan Oct 30, 2023
Collaborator

josharian
Oct 5, 2023
Collaborator