Problem: float values loses precision after SQL transformation

[arajkumar]

Solution: Use format string %.15f which would retain 15 digits after the decimal point.

[arajkumar]

@dimitri I'm not sure this is a right solution, but I wanted to understand why we settled on %f which outputs floating point values with low resolution.?

I had gone through [1][2] and I agree that [1][2] will be the right fix for the problem, I would still like to understand why not just bump up the resolution as proposed in this fix? I can add tests if required.

The test cdc-wal2json seem to fail because of bumping up the resolution, but that is not a problem and I can fix expected file.

[1] eulerto/wal2json#255
[2] ebe3f33 (edited)

[shubhamdhama]

@arajkumar The actual problem is here,

			case JSONNumber:
			{
				double x = json_value_get_number(jsval);

				valueColumn->oid = FLOAT8OID;
				valueColumn->val.float8 = x;
				valueColumn->isNull = false;
				break;
			}

I believe JSON doesn't have limit on the length of numeric numbers. But when it's converted to a parson JSON object it detects its type as number. Hence the precision of numeric data type, which is pretty much endless, is lost when stored in double of C. Long time back I tried to patch parson to return numeric data type as string as well which worked fine (or maybe there is a way out of box in parson to read numeric as string in the first place). But then you realize Postgres supports +/-NaN and +/- Inf which comes as null from wal2json. That would lead to loss of data integrity. I guess you already know this from linked discussion.

That's why for wal2json can't be used reliable without numeric-data-types-as-string so the code path you pointed above is not used practically (and perhaps we can even delete it).

[arajkumar]

Thanks @shubhamdhama.