System.Text.Json: (De)serialization support for quoted numbers

[NickCraver]

Original proposal by @NickCraver (click to view)

Apologies if this issue exists already...I hunted and couldn't find one.

I've been trying to switch a lot of usages over to System.Text.Json from Newtonsoft and Jil, but a recurring theme is handling external APIs and their not-quite-right returns of JSON.

Unfortunately, an example I'm seeing over and over is:

{ "field":"12345" }

They're numbers, but as strings. int, double, decimal, whatever. It happens all the time. And since it's someone's API, it's unlikely we'll get the world to fix this.

Note: this is something Newtonsoft handles by default, which is why I don't think most people realize it's an issue.

Is there any chance we can let System.Text.Json handle number types being quoted when deserializing? Something a user opts-into via JsonSerializerOptions?

Small repro example:

void Main()
{
	JsonSerializer.Parse<Foo>(@"{""Bar"":""1234""}");
}
public class Foo
{
	public int Bar { get; set; }
}

cc @ahsonkhan @steveharter

---

Edited by @layomia:

Users want to be able to deserialize JSON strings into number properties. Scenarios include deserializing "NaN", "Infinity" and "-Infinity" (#31024).

We should add an option to enable this scenario, and possibly allow serializing numbers as strings.

These semantics allow better interop with various API endpoints accross the web.

API Proposal

namespace System.Text.Json
{
    public partial sealed class JsonSerializerOptions
    {
        public JsonNumberHandling NumberHandling { get; set; }
    }
}

namespace System.Text.Json.Serialization
{
    [Flags]
    public enum JsonNumberHandling : byte
    {
        /// <summary>
        /// No specified number handling behavior. Numbers can only be read from <see cref="JsonTokenType.Number"/> and will only be written as JSON numbers (without quotes).
        /// </summary>
        None = 0x0,
        /// <summary>
        /// Numbers can be read from <see cref="JsonTokenType.String"/>. Does not prevent numbers from being read from <see cref="JsonTokenType.Number"/>.
        /// </summary>
        AllowReadingFromString = 0x1,
        /// <summary>
        /// Numbers will be written as JSON strings (with quotes), not as JSON numbers.
        /// </summary>
        WriteAsString = 0x2,
        /// Floating point constants represented as <see cref="JsonTokenType.String"/> tokens
        /// such as "NaN", "Infinity", "-Infinity", can be read when reading, and such CLR values
        /// such as <see cref="float.NaN"/>, <see cref="double.PositiveInfinity"/>, <see cref="float.NegativeInfinity"/> will be written as their corresponding JSON string representations.
        AllowNamedFloatingPointLiterals = 0x4
    }

    public partial sealed class JsonNumberHandlingAttribute : JsonAttribute
    {
        public JsonNumberHandling Handling { get; }
    
        public JsonNumberHandlingAttribute(JsonNumberHandling handling)
        {
            Handling = handling;
        }
    }
}

Usage

Allow reading numbers from strings; write numbers as strings.

public class ClassWithInts
{
    public int NumberOne { get; set; }
    public int NumberTwo { get; set; }
}

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString | JsonNumberHandling.WriteAsString 
};

string json = @"{""Number1"":1,""Number2"":""2""}";

ClassWithInts @class = JsonSerializer.Deserializer<ClassWithInts>(json, options);
Console.WriteLine(@class.NumberOne); // 1
Console.WriteLine(@class.NumberTwo); // 2

json = JsonSerializer.Serialize(@class, options);
Console.WriteLine(json); // @"{""Number1"":""1"",""Number2"":""2""}";

Allow reading numbers from floating point constants; write floating point constants

Given a class:

public class ClassWithNumbers
{
    public int IntNumber { get; set; }
    public float FloatNumber { get; set; }
}

Without the new option, reading floating-point-constant representations fails:

string json = @"{""IntNumber"":1,""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json);

// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Single. Path: $.FloatNumber | LineNumber: 0 | BytePositionInLine: 34.

Writing also fails:

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj);

// Unhandled exception. System.ArgumentException: .NET number values such as positive and negative infinity cannot be written as valid JSON.

With the new option, reading floating-point-constant representations works:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":1,""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // NaN

Writing floating-point-constant representations also works:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":-1,"FloatNumber":"NaN"}

Allow reading numbers from string; support reading and writing floating point constants

Reading:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":""1"",""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // NaN

Writing:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":-1,"FloatNumber":"NaN"}

Write numbers as strings; support reading and writing floating point constants

Reading:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

string json = @"{""IntNumber"":""1""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

Console.WriteLine(obj.IntNumber); // 1
Console.WriteLine(obj.FloatNumber); // 0

string json = @"{""IntNumber"":""1"",""FloatNumber"":""NaN""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);

// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 16.

Writing:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.WriteAsString | JsonNumberHandling.AllowNamedFloatingPointLiterals
};

var obj = new ClassWithNumbers
{
    IntNumber = -1,
    FloatNumber = float.NaN
};

string json = JsonSerializer.Serialize(obj, options);
Console.WriteLine(json); // {"IntNumber":"-1","FloatNumber":"NaN"}

Read semantics are the same as the serializer's default (no quotes)

For example, integers cannot have decimal places:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1.0""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 17.

Similarly, there can't be any leading or trailing trivia:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1a""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 15.

Leading or trailing whitespace is also not allowed:

var options = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString
}

string json = @"{""IntNumber"":""1 ""}";
ClassWithNumbers obj = JsonSerializer.Deserialize<ClassWithNumbers>(json, options);
// Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.IntNumber | LineNumber: 0 | BytePositionInLine: 15.

Note that formats such as the currency and percentage formats are not allowed. Commas (,), the percent symbol (%), and underscores (_) are not permitted in input strings.

Notes

Newtonsoft compat

• Newtonsoft.Json allows implicit conversions from strings to numbers on deserialization, i.e, no custom logic or setting is required. We will require that users explictly opt in to this behavior.
• Newtonsoft.Json does not provide a built-in way to serialize numbers as strings. Custom logic, e.g. a custom converter needs to provided for this behavior. This proposal includes an easy way to specify this behavior.

Read/write semantics

• The number reading and writing behavior of this feature is exactly the same as if no quotes were specified, except that surrounding quotes can be allowed on deserialization, and can be written on serialization.
  • Reading/writing is not culture aware (i.e InvariantCulture). A custom converter has to be specified for culture-aware handling.
  • The "G17" standard numeric format is used when reading and writing double representations.
  • The "G9" standard format is used when reading and writing float representations.
  • Percentage and currency formats are not supported.
• Numbers represented as JsonTokenType.String tokens will be unescaped if needed when deserializing. This is in keeping with other string-based parsing for types like DateTime and Guid.
• Only the literal "NaN", "Infinity", and "-Infinty" values are allowed. This is in keeping with the most common variations on the web. The matching is case-sensitive. We can be more permissive in the future, based on user feedback.

[NickCraver]

From Twitter: a lot of people correctly point out that numbers are intentionally quoted in APIs so they're not assumed to be floats and lose precision. It's an excellent point, and even more reason to support deserializing them I think.

[bartonjs]

numbers are intentionally quoted in APIs so they're not assumed to be floats and lose precision

To me that says that the user should still deserialize them as strings and only turn them into numbers when they're ready to accept the loss of precision.

[manigandham]

@bartonjs

That's the intention. The data types for the class properties are what we want the final result to be.

If we have to use string-based properties first and then parse ourselves then there's not much utility for this serialization library in terms of mapping to types at all, especially for APIs.

[steveharter]

With support for a custom double and\or decimal converter you can do this yourself. Perhaps there will be future support for a set of "Json.NET-compat" converters at some point but that is not available now.

Below is an skeleton of such a custom converter. Also see https://github.com/dotnet/corefx/issues/38713 which has a custom object converter that converts to bool if token type is True\False.

public class DoubleConverterWithStringSupport : JsonConverter<double>
{
    public override double Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            return ...;
        }

        // Default behavior; will throw if TokenType != Number
        return reader.GetDouble();
    }

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        // Write as number or string? Could also use [JsonConverter] to specify a different converter for certain properties.
    }
}

[steveharter]

Moving to Future. cc @rynowak

[NickCraver]

I'm trying to unwrap all that (thanks for the workaround!) - is that saying we'd never consider adding it to the base serializer and it'd only ever live as an extension point? I'm finding the lack of this to be a very common pain point, so I'm wondering what the bar is for inclusion if so.

[bartonjs]

@NickCraver I think it's just "there's a whole lot of policy here, and we would need to completely spec it out, and we're trying to be feature complete 30 days ago... so this isn't required for v1 (.NET Core 3.0)"

• Is there an option to say if deserializing should be strict vs coercive? (Personally, I hate coercive APIs, so I would always want it in "this data better conform to schema" mode)
• If numbers are deserialized through either JSON Number or JSON String, how are they written?
  • Do we need per property toggles other than the converters?
• Is only invariant format supported? Or also CurrentCulture? Or CurrentUICulture?
• How should the deserialization handle it if the property is typed as int but the value is the string "3.0"? Is that 3, or an exception? (Currently (IIRC) if it's a JSON number that says 3.0 and the field is int it throws)
• Should it ignore trailing spaces if they ended up in the string? Leading spaces?
• Should it support the percentage format? The currency format?

It's a simple-sounding request, but there are a lot of details and many of them are hard to change once the feature first ships. So the .NET Core 3.0 answer is "deserialization doesn't coerce numbers out of JSON strings; but the converters feature lets you effectively opt-in on a property-by-property basis".

[NickCraver]

Totally understand, I was mostly curious about where the bar is and if moving to this was a good from a future standpoint (or we shouldn't try to move certain code sections to it). That list helps a lot - I agree this can't be just tossed in.

Thanks for the quick reply!

[manigandham]

The converter option mentioned by @steveharter works great now in the .NET Core 3.0 preview-7 release. Confirmed server-side and in Blazor.

public class LongToStringConverter : JsonConverter<long>
{
    public override long Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            ReadOnlySpan<byte> span = reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan;
            if (Utf8Parser.TryParse(span, out long number, out int bytesConsumed) && span.Length == bytesConsumed)
                return number;

            if (Int64.TryParse(reader.GetString(), out number))
                return number;
        }

        return reader.GetInt64();
    }

    public override void Write(Utf8JsonWriter writer, long value, JsonSerializerOptions options)
    {
        writer.WriteStringValue(value.ToString());
    }
}

services.AddControllersWithViews()
   .AddJsonOptions(options =>
   {
        options.JsonSerializerOptions.Converters.Add(new LongToStringSupport());
   });

This was developed in the other thread: https://github.com/dotnet/corefx/issues/36563#issuecomment-519788518

[manigandham]

@AlexPaskhin

It's not every type... this case is about Javascript/JSON not supporting large numeric types so it needs a string conversion, and the reasoning by the team on why this isn't built-in makes sense. The converter code is right there so you copy/paste if you need it, but I guess they can add some default converters for long and double in the package.

What other problems are you having? You don't have to use System.Text.Json if you need more flexibility, Json.NET is still available and easily enabled.

[oleg-varlamov]

We have a similar situation: we decided to move our project to .net core 3.0. and use the built-in System.Text.Json. We had to rewrite a small part of the functionality, but in general, the build was successful.
However, in runtime we began to receive json parsing errors(not only from other services, but also from our own). We had to do a commit revert and return to Newtonsoft.Json.
Of course, it is clear that it is necessary to correctly transfer all numeric values, without quotes. But our partners are unlikely to want to redo the already approved integration format.

[steveharter]

For 5.0 we can consider an option to support "looser" deserialization modes depending on how common these scenarios are, and whether we can agree on common semantics.

However I believe all such modes can be supported today by using custom converters and by extending JsonConverterAttribute so that specific properties can be targeted (instead of all properties). For some cases it may be that there is no common consensus for the semantics so leaving it as-is and requiring the use of custom converters and custom attributes is the best solution. Perhaps doc and examples for common scenarios works too, or the community creates extensions for common scenarios.

A 5.0 feature could be to add a options.DeserializeQuotedNumbers == true. Then we need to determine if serializing as numbers is always desired, otherwise we'd need another option like SerializeNumbersAsQuoted (perhaps a flags enum would be easier to extend instead of separate option properties). There are subtleties here like do we account for culture settings? We'd also need a new attribute so that specific properties can be targeted instead of all through the global option.

Similarly, there has been feedback on what primitive types to deserialize when the property is a System.Object. Today that is JsonElement. The requests have been to deserialize true\false as System.Bool, numbers as System.Double and strings as System.String instead of JsonElement.

Adding additional option(s) for this is also possible, however there are subtleties: precision can be lost (e.g. System.Double has less precision than System.Decimal -- what is the intent?) and when the JSON values belong to an object being deserialized into System.Object, the deserializer would still need to deserialize the object and all of its properties as JsonElement since it doesn't know what type to create and deserialize into (without getting into polymorphic deserialization, which is a separate feature).

[ericstj]

From @dansiegel in dotnet/corefx#42396

There should either be a built in JsonConverter that can handle this, a Serialization option to allow handling this, or it should just be smart enough to try to convert from string to any given numeric type.

[Going-Gone]

Maybe i'm mistaken, but we shouldn't have to have a custom converter.
Is it possible to mimic what the JsonConvert class does? This would eliminate the burden of migrating to this api, and would be a big QOL improvement. It also would make more sense for people to migrate then.

Int32
Int64
DateTime formats

[dansiegel]

I would hate to think I would have to decorate all of my properties to handle this. Really the built in converter should be smart enough to try to parse the string to the given numeric type.

[bartonjs]

Really the built in converter should be smart enough to try to parse the string to the given numeric type.

So long as it's an option. I definitely don't want it coercing values in my payloads. If I said I want a number, thou shalt not send me a string.

[AlexPaskhin]

Just don’t forget that JSON (JavaScript Object Notation) is a lightweight data-interchange format. So, JavaScript doesn’t have types. So, If you are going to communicate with “JavaScript” services, Angular, React, TypeScript, Java (currently ~99%) you have to use Newtonsoft json. Unfortunately System.Text.Json doesn’t like “JavaScript” typeless word , cannot find proper type mapping, … and throws exceptions. Unfortunately, it was not developed the “JavaScript” version of System.Text.Json, therefor you can use it (System.Text.Json) in communication only (System.Text.Json) <-> (System.Text.Json) only.

[ghost]

Please add this option
options.DeserializeQuotedNumbers == true
and allow it to deserialize to decimal type.
"mynum":"0.82" -> decimal type.

[layomia]

From @IgorMenshikov in #725

I am migrating to .NET Core 3.1 and have a class with null-able values (decimal, DateTime) and want deserialize JSON to it. I received JSON from a client with property values as strings. Here is the sample code:

public class Range
{
    public decimal? Start { get; set; }
    public decimal? End { get; set; }
}

static void Main(string[] args)
{
    var json = "{\"Start\":\"1\",\"End\":\"2\"}";
    var jsonNumber = "{\"Start\": 1,\"End\": 2}";

    var j1 = Newtonsoft.Json.JsonConvert.DeserializeObject<Range>(json); // works fine
    j1.ToString();

    var jn = System.Text.Json.JsonSerializer.Deserialize<Range>(jsonNumber); // works fine
    jn.ToString();

    var js = System.Text.Json.JsonSerializer.Deserialize<Range>(json); // exception
    js.ToString();
}

This code throws an error:

'The JSON value could not be converted to System.Nullable`1[System.Decimal].

I have tried to use a workarounf at:
dotnet/corefx#41070 (comment)

but it does not help. Can something be some with that? Any workaround? Newtonsoft can handle that without problems.

[layomia]

From @IgorMenshikov in dotnet/runtime#725

(De)serialization support for quoted numbers should also work for nullable types.

[layomia]

Per https://github.com/dotnet/corefx/issues/41442, this issue should consider the deserialization of "Infinity", "-Infinity" and "NaN" into double types (double.PositiveInfinity, double.NegativeInfinity, double.NaN), and vice-versa.

From @pranavkm in https://github.com/dotnet/corefx/issues/41442:

From @valeriob on Monday, September 30, 2019 12:49:01 PM

Hi,
we are upgrading some applications to aspnetcore 3.0, we have been bitten by something unexpected : System.Text.Json refuse to serialize double.NaN values.
This cause really nasty bugs because something that works suddenly does not because some result of some calculation is different, or some data changes in some way.
I see that the recommendation (https://docs.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.0&tabs=visual-studio#jsonnet-support) is to use Json.Net.

But how can ppl use this library if such a simple and common use case is not covered ?
I do not know any application that can live without this feature, i understand the fact that there is a specification, but it may very well be unpractical.
https://thefactotum.xyz/post/the-devil-is-in-the-json-details/
Python, Go, Javascript,Rust, Ruby handle it without making much fuss 😄

Thanks
Valerio

Copied from original issue: aspnet/AspNetCore#14571

[layomia]

Renaming this issue to explicitly mark that serialization of quoted numbers is being considered here as well, per @steveharter's comment above:

A 5.0 feature could be to add a options.DeserializeQuotedNumbers == true. Then we need to determine if serializing as numbers is always desired, otherwise we'd need another option like SerializeNumbersAsQuoted (perhaps a flags enum would be easier to extend instead of separate option properties). There are subtleties here like do we account for culture settings? We'd also need a new attribute so that specific properties can be targeted instead of all through the global option.

[layomia]

If added, (de)serialization support for numbers as dictionary keys (https://github.com/dotnet/corefx/issues/40120) will use semantics defined here.

[danchaseCTS]

Isn't JSON a spec? I see a lot of requests that are app specific, that would be solved simply by having the app specify the value properly.

For example, Angular wants to put something like:
Amount: "15.50"
unless you make the input type="number", in which it will now put:
Amount: 15.50

As soon as you start converting things you will experience feature bloat and bugs like no other, actually creating pain instead of helping future roadmaps for any apps that use the library. JSON isn't meant to be a text processor.

[terrajobst]

Video

• Looks good.
• We should have a discussion around defaults for .NET 5, especially web.
• We decided to rename JsonNumberHandling.None an JsonNumberHandling.Strict

namespace System.Text.Json
{
    public partial sealed class JsonSerializerOptions
    {
        public JsonNumberHandling NumberHandling { get; set; }
    }
}
namespace System.Text.Json.Serialization
{
    [Flags]
    public enum JsonNumberHandling
    {
        Strict = 0x0,
        AllowReadingFromString = 0x1,
        WriteAsString = 0x2,
        AllowNamedFloatingPointLiterals = 0x4
    }

    [AttributeUsage(
        AttributeTargets.Class |
        AttributeTargets.Struct |
        AttributeTargets.Property |
        AttributeTargets.Field, AllowMultiple = false)]
    public partial sealed class JsonNumberHandlingAttribute : JsonAttribute
    {
        public JsonNumberHandlingAttribute(JsonNumberHandling handling);
        public JsonNumberHandling Handling { get; }    
    }
}

[paulhickman-a365]

I'd like to this proposal include an overload on Utf8JsonWriter.WriteNumberValue to specify if it should be quoted or not. e.g.

int x= 5;
writer.WriteNumberValue(x,true) // writes "5"
writer.WriteNumberValue(x,false) // writes 5

This avoids having to do the extra memory allocation in

writer.WriteStringValue(x.ToString()) // writes "5"

and it allows you to write efficient code when the JSON schema you have to comply with has some numbers quoted but not all of them.

[layomia]

@paulhickman-a365 this issue addresses support at the serializer level only. Please open a new issue with the "API proposal" format for your suggestion for the writer to be considered.

[ironpython2001]

It seems this is a missing feature. Better to add this feature like newtonsoft.json

[layomia]

@ironpython2001 this feature is checked in and coming in .NET 5. You can try it in .NET 5 RC1 - https://devblogs.microsoft.com/dotnet/announcing-net-5-0-rc-1/.