Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JsonSerializer support for fields & non-public accessors #34558

Closed
layomia opened this issue Apr 5, 2020 · 5 comments
Closed

JsonSerializer support for fields & non-public accessors #34558

layomia opened this issue Apr 5, 2020 · 5 comments
Assignees
Labels
api-approved API was approved in API review, it can be implemented area-System.Text.Json
Milestone

Comments

@layomia
Copy link
Contributor

layomia commented Apr 5, 2020

For streamlined API review and due to common API suggestions, this issue addresses non-public accessor support, and field support.

Motivation

Using non-public accessors for serialization and deserialization

From #29743:

Non-public setters for public properties are not used when deserializing. Other serializers have this feature. As a workaround, user's have to write custom converter's for such POCOs, which isn't trivial in case of a complex object. This feature provides an opt-in for the serializer to use non-public setters.

Enabling non-public getter usage is included for parity with Newtonsoft.Json which supports this when the JsonProperty attribute is used, and to prevent complicating the API surface if this is ever desired in the future.

This feature was scoped out of v1 (3.x), as the objective was to support simple POCOs. We elected to make this feature opt-in due to security concerns with non-public support.

This feature does not include support for non-public properties.

A related feature that allows deserialization of immutable objects through parameterized constructors was recently added in #33444.

Field support

From #876:

There is no way to serialize and deserialize fields using JsonSerializer.

While public fields are generally not recommended, they are used in .NET itself (see value tuples) and by users.

This feature was scoped out of v1 (3.x) due to lack of time, as we prioritized supporting simple POCOs with public properties.

We elected to have an opt-in model for field support because it would be a breaking change to support them by default, and also because public fields are generally not recommended. Other serializers, including Newtonsoft.Json, Utf8Json, and Jil, support this feature by default.

API proposal

Option 1

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether fields are included when serializing and deserializing.
        /// The default value is <see langword="false"/>.
        /// </summary>
        /// <remarks>
        /// Only public fields will be serialized and deserialized.
        /// </remarks>
        public bool IncludeFields { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Provides options for how object members should be serialized and deserialized.
    /// </summary>
    [AttributeUsage(AttributeTargets.Class | Attributes.Struct, AllowMultiple = false)]
    public partial sealed class JsonObjectAttribute : JsonAttribute
    {
        /// <summary>
        /// Indicates whether non-public property getters and setters are ignored when serializing and deserializing.
        /// </summary>
        /// <remarks>
        /// The default value is <see langword="false"/>.
        /// <see cref="JsonIgnoreAttribute"> can be used to ignore individual properties.
        /// </remarks>
        public bool IgnoreNonPublicAccessors { get; set; }

        /// <summary>
        /// Indicates whether public fields are ignored when serializing and deserializing.
        /// </summary>
        /// <remarks>
        /// The default value is <see langword="false"/>.
        /// <see cref="JsonIgnoreAttribute"> can be used to ignore individual fields.
        /// </remarks>
        public bool IgnoreFields { get; set; }

        /// <summary>
        /// Initializes a new instance of <see cref="JsonObjectAttribute"/>.
        /// </summary>
        public JsonObjectAttribute() { }
    }

    /// <summary>
    /// When applied to public properties, non-public getters or setters will be used when serializing and deserializing.
    /// When applied to public fields, they will be included when serializing and deserializing.
    /// </summary>
    [AttributeUsage(AttributeTargets.Property | Attributes.Field, AllowMultiple = false)]
    public partial sealed class JsonMemberAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonMemberAttribute"/>.
        /// </summary>
        public JsonMemberAttribute() { }
    }
}

Usage

Including fields and using non-public accessors

Given a Person class:

[JsonObject]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonMember]
    public string FirstName;

    [JsonMember]
    public string LastName;

    [JsonMember]
    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 123

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":123}
Including fields and ignoring non-public accessors

Given a Person class:

[JsonObject(IgnoreNonPublicAccessors = true)]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonMember]
    public string FirstName;

    [JsonMember]
    public string LastName;

    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 0

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":0}
Including fields globally but ignoring them for a particular type

Given an Account class:

public class Account
{
    public int Id;

    public AccountType Type;

    public Person Owner;
}

public enum AccountType
{
    Checking = 0,
    Saving = 1,
    MoneyMarket = 2
}

And a Person class:

[JsonObject(IgnoreFields = true)]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Account instances can be serialized and deserialized:

JsonSerializerOptions options = new JsonSerializerOptions
{
    IncludeFields = true
};

string json = @"{
    ""Id"": 12345,
    ""Type"": 1,
    ""Owner"": {
        ""FirstName"":""Jet"",
        ""LastName"":""Doe"",
        ""Id"":123
    }
}";

Account account = JsonSerializer.Deserialize<Account>(json, options);
Console.WriteLine(account.Id); // 12345
Console.WriteLine(account.Type); // Saving

Person person = account.Person;
Console.WriteLine(person.FirstName) // null
Console.WriteLine(person.LastName) // null
Console.WriteLine(person.Id) // 123

// Prepare for serialization.
person.FirstName = "Jet";
person.LastName = "Doe";

json = JsonSerializer.Serialize(account, options);
Console.WriteLine(json); // {"Id":12345,"Type":1,"Owner":{"Id":123}}

Option 2

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        /// <summary>
        /// Determines whether fields are included when serializing and deserializing.
        /// The default value is false.
        /// </summary>
        /// <remarks>
        /// Only public fields will be serialized and deserialized.
        /// </remarks>
        public bool IncludeFields { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// When applied to types, non-public getters or setters will be used when serializing and deserializing all public properties.
    /// Use <see cref="JsonIgnoreAttribute"/> to opt-out.
    /// When applied to public properties, non-public getters or setters will be used when serializing and deserializing.
    /// </summary>
    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Property, AllowMultiple = false)]
    public sealed class JsonPropertySerializableAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonPropertySerializableAttribute"/>.
        /// </summary>
        public JsonPropertySerializableAttribute() { }
    }

    /// <summary>
    /// When applied to a type, all its public fields will be serialized and deserialized. Use <see cref="JsonIgnoreAttribute"/> to opt-out.
    /// When applied to a public field, it will be serialized and deserialized.
    /// </summary>
    /// <remarks>
    /// The absence of this attribute on a class, struct, or field, does not preclude the fields from serialization or deserialization, if <see cref="JsonSerializerOptions.IncludeFields"> is set to <see langword="true">.
    /// </remarks>
    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Field, AllowMultiple = false)]
    public sealed class JsonFieldSerializableAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonFieldSerializableAttribute"/>.
        /// </summary>
        public JsonFieldSerializableAttribute() { }
    }
}

Usage

Including fields and using non-public accessors

Given a Person class:

[JsonPropertySerializable]
[JsonFieldSerializable]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonPropertySerializable]
    public string FirstName;

    [JsonPropertySerializable]
    public string LastName;

    [JsonFieldSerializable]
    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 123

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":123}
Including fields and ignoring non-public accessors

Given a Person class:

[JsonFieldSerializable]
public class Person
{
    public string FirstName;
    
    public string LastName;

    public int Id { get; private set; }
}

Alternatively,

public class Person
{
    [JsonFieldSerializable]
    public string FirstName;

    [JsonFieldSerializable]
    public string LastName;

    public int Id { get; private set; }

An instance can be serialized or deserialized:

string json = @"{""FirstName"":""Jet"",""LastName"":""Doe"",""Id"":123}";

Person person = JsonSerializer.Deserialize<Person>(json);
Console.WriteLine(person.FirstName); // Jet
Console.WriteLine(person.LastName); // Doe
Console.WriteLine(person.Id); // 0

json = JsonSerializer.Serialize(person);
Console.WriteLine(json); // {"FirstName":"Jet","LastName":""Doe","Id":0}
Including fields globally but ignoring them for a particular type

Given an Account class:

public class Account
{
    public int Id;

    public AccountType Type;

    public Person Owner;
}

public enum AccountType
{
    Checking = 0,
    Saving = 1,
    MoneyMarket = 2
}

And a PersonClass:

public class Person
{
    [JsonIgnore]
    public string FirstName;
    
    [JsonIgnore]
    public string LastName;

    [JsonFieldSerializable]
    public int Id { get; private set; }
}

Alternatively,

[JsonFieldSerializable]
public class Person
{
    [JsonIgnore]
    public string FirstName;
    
    [JsonIgnore]
    public string LastName;

    public int Id { get; private set; }
}

Account instances can be serialized and deserialized:

JsonSerializerOptions options = new JsonSerializerOptions
{
    IncludeFields = true
};

string json = @"{
    ""Id"": 12345,
    ""Type"": 1,
    ""Owner"": {
        ""FirstName"":""Jet"",
        ""LastName"":""Doe"",
        ""Id"":123
    }
}";

Account account = JsonSerializer.Deserialize<Account>(json, options);
Console.WriteLine(account.Id); // 12345
Console.WriteLine(account.Type); // Saving

Person person = account.Person;
Console.WriteLine(person.FirstName) // null
Console.WriteLine(person.LastName) // null
Console.WriteLine(person.Id) // 123

// Prepare for serialization.
person.FirstName = "Jet";
person.LastName = "Doe";

json = JsonSerializer.Serialize(account, options);
Console.WriteLine(json); // {"Id":12345,"Type":1,"Owner":{"Id":123}}

To override JsonSerializer.IncludeFields when set to true, [JsonIgnore] would need to be set on each field to be ignored.

Notes

  • The opt-in mechanism for non-public accessor is per property and per type, not "globally" on JsonSerializerOptions. This is to prevent non-public member access on types that are not owned by the user.

  • Features involving the the use of non-public accessors, including constructors, getters, setters etc. may not be supported in the upcoming AOT/code-gen work due to the likely need to use runtime reflection for those scenarios.

  • The options on JsonSerializerOptions that apply to properties will also apply to fields. This is intuitive and keeps the API surface clean:

    • IgnoreNullValues
    • PropertyNameCaseInsensitive
    • PropertyNamingPolicy
  • Existing attributes that apply to properties will also apply to fields:

    • JsonConverterAttribute
    • JsonExtensionDataAttribute
    • JsonIgnoreAttribute
    • JsonPropertyNameAttribute
  • Interfaces will not be added as a target until the implications for extended polymorphism support are understood. This is in keeping with JsonConverterAttribute where an interface is not a valid target: Add AttributeTargets.Interface to JsonConverterAttribute #33112.

  • As with public properties, public fields may also bind with constructor parameters during deserialization, with the same semantics.

@layomia layomia added api-ready-for-review area-System.Text.Json blocking Marks issues that we want to fast track in order to unblock other important work labels Apr 5, 2020
@layomia layomia added this to the 5.0 milestone Apr 5, 2020
@layomia layomia self-assigned this Apr 5, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Apr 5, 2020
@layomia layomia removed the untriaged New issue has not been triaged by the area owner label Apr 5, 2020
@Symbai
Copy link

Symbai commented Apr 5, 2020

Features involving the the use of non-public accessors, including constructors, getters, setters etc. may not be supported in the upcoming AOT/code-gen work due to the likely need to use runtime reflection for those scenarios.

Why would it require runtime reflection and what speaks against it (Performance? Security?) and what speaks against it making it optional so in case of performance developers have a choice when performance / security isn't important?

@layomia
Copy link
Contributor Author

layomia commented Apr 5, 2020

Why would it require runtime reflection

In a design where serializers/converters are generated at build-time to reduce the serializer's start-up cost and reduce/eliminate the usage of reflection features that are not supported by some platforms, non-public usage may not be supported dependent on the output assembly of the generated code. This may force the serializer down existing code paths which employ such reflection features as a fallback, which may be problematic depending on the scenario/platform.

what speaks against it making it optional so in case of performance developers have a choice when performance / security isn't important?

This is a valid design possibility.

These are very early thoughts on potential serializer code-gen work in the future; it is not clear yet how non-public usage may affect this work.

@terrajobst terrajobst added api-approved API was approved in API review, it can be implemented and removed api-ready-for-review blocking Marks issues that we want to fast track in order to unblock other important work labels Apr 7, 2020
@terrajobst
Copy link
Member

terrajobst commented Apr 7, 2020

Video

  • Let's rename JsonMemberAttribute to JsonIncludeAttribute
  • We should change JsonIgnoreAttribute to include fields
  • We don't see a need for JsonObjectAttribute it should be a per property/field decisions
  • We should treat read-only fields like get-only properties
  • We can decide to support private/internal fields/properties by applying JsonInclude, but we need think through implications for AOT (generating the serialzier as part of the type would make this possible tho)
namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public bool IncludeFields { get; set; }
    }
}
namespace System.Text.Json.Serialization
{
    [AttributeUsage(AttributeTargets.Property |
                    Attributes.Field, AllowMultiple = false)]
    public sealed class JsonIncludeAttribute : JsonAttribute
    {
        public JsonIncludeAttribute();
    }
}

@layomia
Copy link
Contributor Author

layomia commented Apr 13, 2020

Updated proposal approved offline:

namespace System.Text.Json
{
    public partial class JsonSerializerOptions
    {
        public bool IgnoreReadOnlyFields { get; set; }
        public bool IncludeFields { get; set; }
    }
}
namespace System.Text.Json.Serialization
{
    [AttributeUsage(AttributeTargets.Property |
                    Attributes.Field, AllowMultiple = false)]
    public sealed class JsonIncludeAttribute : JsonAttribute
    {
        public JsonIncludeAttribute();
    }
}

IgnoreReadOnlyFields was added for parity with IgnoreReadOnlyProperties.


The serializer's defintion for "read only" is as follows:

  • for properties, "read only" means has a public getter but no public setter.
  • for fields, "read only" means FieldInfo.IsInitOnly is true.

This definition may be updated following the upcoming C# lang init-only properties (& perhaps fields), dependent on the type/reflection metadata made available.

If init-only setters are available and public, it would be reasonable for the serializer to use them for deserialization. We would have to evaluate whether the serialization of such properties & fields should be influenced by options.IgnoreReadOnly[Properties | Fields]. Any change here may be breaking.

cc @terrajobst

@layomia layomia changed the title JsonSerializer support for non-public accessors and fields JsonSerializer support for fields & non-public accessors Apr 17, 2020
@layomia
Copy link
Contributor Author

layomia commented May 11, 2020

Closing this composite issue as there are tracking issues for the sub-tasks:

@layomia layomia closed this as completed May 11, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
api-approved API was approved in API review, it can be implemented area-System.Text.Json
Projects
None yet
Development

No branches or pull requests

4 participants