Inconsistent rounding of significant digits

[drosen0]

Background

• Common rounding rules say simply that you round up if the digit is 5 or above.
• Scientific rounding rules are a little more complex. These rules are defined in ASTM E29 sections 6.4.1 through 6.4.4.

While I wouldn't expect d3-format to support scientific rounding rules, I would expect that it would apply a consistent set of rules. I know that this is a difficult problem to solve, partly due to the binary representation of floating point numbers.

Test Cases

All of these examples use the format function returned by

d3.format('.3r')

All of the x.45500 cases return the common rounding result:

Input	Output	Common	Scientific
1.45500	1.46	1.46✔️	1.45
2.45500	2.46	2.46✔️	2.45
3.45500	3.46	3.46✔️	3.45
4.45500	4.46	4.46✔️	4.45
5.45500	5.46	5.46✔️	5.45
6.45500	6.46	6.46✔️	6.45
7.45500	7.46	7.46✔️	7.45
8.45500	8.46	8.46✔️	8.45
9.45500	9.46	9.46✔️	9.45

The x.55500 cases return a mix of common and scientific rounding results:

Input	Output	Common	Scientific
1.55500	1.55	1.56	1.55✔️
2.55500	2.56	2.56✔️	2.55
3.55500	3.56	3.56✔️	3.55
4.55500	4.55	4.56	4.55✔️
5.55500	5.55	5.56	5.55✔️
6.55500	6.55	6.56	6.55✔️
7.55500	7.55	7.56	7.55✔️
8.55500	8.55	8.56	8.55✔️
9.55500	9.55	9.56	9.55✔️

All of the x.55501 cases return the correct result, which is the same for common and scientific rounding methods:

Input	Output	Common	Scientific
1.55501	1.56	1.56✔️	1.56✔️
2.55501	2.56	2.56✔️	2.56✔️
3.55501	3.56	3.56✔️	3.56✔️
4.55501	4.56	4.56✔️	4.56✔️
5.55501	5.56	5.56✔️	5.56✔️
6.55501	6.56	6.56✔️	6.56✔️
7.55501	7.56	7.56✔️	7.56✔️
8.55501	8.56	8.56✔️	8.56✔️
9.55501	9.56	9.56✔️	9.56✔️

[mbostock]

The values you are using cannot be represented exactly by JavaScript numbers due to the nature of IEEE 754 floating point. More precise representations of the values you are using, as computed by number.toFixed(20), are:

• 1.55500 ↦ 1.55499999999999993783
• 2.55500 ↦ 2.55500000000000015987
• 3.55500 ↦ 3.55500000000000015987
• 4.55500 ↦ 4.55499999999999971578
• 5.55500 ↦ 5.55499999999999971578
• 6.55500 ↦ 6.55499999999999971578
• 7.55500 ↦ 7.55499999999999971578
• 8.55500 ↦ 8.55499999999999971578
• 9.55500 ↦ 9.55499999999999971578

As you can see, the two values that you consider to exhibit inconsistent behavior in D3 are the two values that are greater than the desired exact values, while the other values are all slightly less than the desired exact values.

For more on this topic, see http://0.30000000000000004.com/.

[drosen0]

But String(1.55499999999999993783) returns 1.555, as does toPrecision(16). JavaScript's String<->Number coercion understands that double precision floating point is only accurate to a precision of 16 digits.

Here's a modified formatDecimal.js that makes the results of r formatting consistent (with common rounding) up to .15r. If you'd consider implementing this sort of a fix, I'll happily submit a PR that also fixes e, f, and g.

// Computes the decimal coefficient and exponent of the specified number x with
// significant digits p, where x is positive and p is in [1, 21] or undefined.
// For example, formatDecimal(1.23) returns ["123", 0].
export default function(x, p) {
  if (x == null || isNaN(x) || x === Infinity || x === -Infinity) { return null; }

  var [mantissa, exponent='0'] = String(x).split('e');
  exponent = +exponent;
  var [whole, fractional=''] = mantissa.split('.');
  var digits = whole + fractional;

  digits = digits.replace(/^0+/, function (match) {
    exponent -= match.length;
    return '';
  });

  if (digits.length < p) {
    digits += Array(p - digits.length + 1).join('0');
  }

  digits = digits.slice(0, p) + '.' + digits.slice(p);
  digits = String(Math.round(digits));

  if (digits === '0') {
    digits = Array(p).join('0');
  }

  return [
    digits,
    exponent + whole.length - 1
  ];
}

[mbostock]

You don’t need to coerce to a string to see that JavaScript (IEEE 754) considers 1.555 and 1.55499999999999993783 to be equal. But that does not mean that mathematically the value is exactly 1.555. If you read the ECMAScript specification, you will see that JavaScript uses the minimum number of digits to uniquely identify a value, and so since these two values are the same from the context of floating point, the shorter representation is chosen. But this is arguably misleading in this context because it gives you a less accurate representation that using 20 digits. (Though naturally since IEEE 754 is binary and not decimal, 20 digits are not enough for an exact representation, either.)

So, your implementation effectively performs rounding twice, first by coercing to a string and then according to the desired behavior in D3. That means that you are treating the value 1.555 as exactly 1.555 (because this is the shortest unique decimal representation in IEEE 754) rather than its natural value of 1.55499999999999993783. Normally I would say this is bad in the context of mathematical operations, but perhaps it makes sense in the context of formatting values? It’s hard to say.

[mbostock]

Here’s further evidence that changing this behavior would be bad. Look at the native behavior of number.toPrecision:

(0.555).toPrecision(2) // "0.56" ⚠️
(1.555).toPrecision(3) // "1.55"
(2.555).toPrecision(3) // "2.56" ⚠️
(3.555).toPrecision(3) // "3.56" ⚠️
(4.555).toPrecision(3) // "4.55"
(5.555).toPrecision(3) // "5.55"
(6.555).toPrecision(3) // "6.55"
(7.555).toPrecision(3) // "7.55"
(8.555).toPrecision(3) // "8.55"
(9.555).toPrecision(3) // "9.55"

[mbostock]

This is related to d3/d3-path#10 (comment).

There are two valid approaches to decimal rounding. The first approach treats the input value x as its “natural” value in binary floating point. The second approach treats the input value x as the shortest-equivalent decimal value, as in the mathematical quantity represented by the result of calling number.toString.

The built-in methods number.toFixed and number.toPrecision chose the first approach, so the result of +(1.555).toFixed(2) is 1.55 because 1.555 is more precisely represented as 1.55499999999999993783 in binary floating point, which is less than 1.555, and thus rounds down.

If you instead interpret 1.555 as its shortest-equivalent decimal value, then it represents exactly 1.555, and thus should round up. And furthermore, it is possible to implement this technique in JavaScript, but it requires string concatentation: +(Math.round(1.555 + "e2") + "e-2") is 1.56. (See also round10.)

As @drosen0 stated, rounding should behave consistently. Applying decimal rounding rules to binary floating point values is perhaps surprising, but it is self-consistent (assuming there’s no implementation bug, which there does not appear to be), and furthermore is consistent with JavaScript’s built-in methods such as number.toFixed and number.toPrecision.

[pietersv]

Does any one know if there is a fork of d3.format that uses Math.round? I love the expressive range of d3.format syntax and its wide use in leading libraries, and get why it is the way it is now. One use of the library is to format numbers for human consumption, who would expect $8.995 to round to $9.99 either simply (ties-round-up) or scientific round-to-even. It's harder to explain that $8.995 is numerically the same as $8.9949999999999992 and thus should round down.