proposal: encoding/json supports Marshaler/Unmarshaler interfaces with a context.Context

[sebastien-rosset]

Proposing to pass a context.Context as an argument to new json.MarshalerXXX and json.UnmarshalerXXX interfaces:

MarshalJSONXXXX(ctx context.Context) ([]byte, error)

The interface specifies a MarshalJsonXXX method that takes a context.Context argument. With a context.Context, custom marshalers can access the context and determine how to marshal the objects at run-time.

Use cases:
Many REST APIs include meta-data that controls which attributes should be included and omitted in the HTTP response body. These controls can be static, meaning the serialization behavior is specified at design time and does not change at runtime. For example, some attributes may be declared as read-only, read-write, read-on-create, write-only, create-only or no-access. A write-only property would be sent by a client in the POST request, and it would not be serialized in the response.

In other cases, the serialization behavior is dynamic and determined at runtime, based on some contextual information. For example, the OData $select operator is used by clients to specify which attributes should be returned in the response body, and other attributes are supposed to be omitted, which is useful for IoT and mobile applications. As another example, regulations and security policies may specify that some attributes must be masked or anonymized. The data masking and anonymization behavior may depend on the data sensitivity level, the role of the user accessing the data, and other runtime attributes. Specific examples of sensitive data are credit card numbers, IP addresses, serial numbers, and Personally-Identifiable Information. Furthermore, in a SaaS environment, the same API may be used by multiple end-users, and each end-user has the ability to assign a sensitivity level to various attributes.

Solutions with existing go runtime:
There are multiple approaches to solve these use cases. For static use cases, one approach is to create separate Go structs for each desired serialization output (HTTP request, HTTP response, data serialization to back-end services, etc). It is also possible to create multiple REST paths, each path providing different levels of access to the data, and assigning authorization rules, but this is still fairly static and must be determined at build time. For dynamic use cases, there is a combinatorial effect, so it's not practical to create separate go structs for each possible combination. Other serialization approaches include the use of reflection, manipulating JSON documents with map[string]interface{}, or creating an enhanced implementation of encoding/json that provides support for more customization.

[as]

I dislike the way context.Context is being used here. Most marshaling options are not request scoped, and they never leave the client in this case. Can you provide an example of how using your proposed interface solves your problem?

[dsnet]

Edit: Disregard this post. I must have been sleepy when I read your post the first time.

Cancelation makes the most sense when the operation is potentially blocking. Most Marshal and Unmarshal operations are entirely in-memory operations (it's possible that they make some blocking calls, but that is very rare). Adding a Context is a tax that everyone has to pay, and it's not a benefit most people will get.

#20280 suggested adding Context to io.Reader and friends. Those methods actually do block and make more sense to take in a context. However, many people are (understandably) opposed to complicating the API when the context is not always useful.

[sebastien-rosset]

@as : a HTTP handler in a go program marshals a JSON document in response to an HTTP request. To give two examples: 1) the JSON document contains some attributes that are localized (i18n); 2) some attributes need to be masked either entirely or partially, due to some per-user data masking policy. The context carries the i18n accepted language and the user information from which data masking policy is derived. There are of course multiple ways to solve this, but we found other methods required a significant amount of glue code.

[rsc]

Contexts are not for per-operation options. It's for timeouts and more global things like authorization credentials, not fine-grained per-call options. We pulled sql.TxOption out of context before Go 1.9 for the same reason. Think about "would this be appropriate to set and force into all operations during a given client RPC", even nested operations calling into unrelated different APIs in back ends (to gather information for the response). Unfortunately I don't know of a good post explaining this nuance with context.

More generally, the json package is almost completely done. If you need significantly extended semantics (and it seems you do), then it would be fine to copy the current one and make your own modifications, creating a new package customized to your use case.