-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add serialization and deserialization of numerals larger than Number.MAX_SAFE_INTEGER
#544
Conversation
0e3a5b4
to
e792722
Compare
Let's hold on merging this for a little while. |
We can do a patch release of the js client right now if that would help
…On Tue, Jul 11, 2023, 12:08 PM Miki ***@***.***> wrote:
Let's hold on merging this for a little while.
—
Reply to this email directly, view it on GitHub
<#544 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA5PRLSZTEDJVPC3ZB47UJDXPV3BNANCNFSM6AAAAAA2FPHO4U>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
74f83ec
to
142cf98
Compare
142cf98
to
d54917a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you write this code from scratch? :)
- Is there really no BOM character that can never be present in the serialized data?
- Where is this serialization used? I think we should have at least one higher level test that shows it.
/* The characters with a highly unlikely chance of occurrence in strings, alone or in combination. */ | ||
_bigIntMarkChars = ['෴', '߷', '֍']; | ||
|
||
/* Generates an array of all combinations of `_bigIntMarkChars` with the requested length. */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know these are unlikely, but is there a more reliable way to have a BOM?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't just use these guys. It creates a combination of 1 to Infinity of them. So even if these guys are all present, there is always a sequence that is long enough to not be present.
A future enhancement could dynamically generate a large collection based on ranges and experiment with them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm probably slow. These are all digits. You get things like ෴18014398509481982
, why not N:18014398509481982
?
d54917a
to
e5d4dc9
Compare
Yes :)
A string containing serialized JSON is still a JS string so all chars that can be used in JS strings can also be present in the serialized JSON. PS, with the exception of U+2028 and U+2029, anything that goes into JSON, goes into JS, but then again, all of this serializing and deserializing is done in JS.
Transport.js uses it to produce the body of requests. It is also exported to be used by others. I will add an integration test. |
e5d4dc9
to
1ac106c
Compare
….MAX_SAFE_INTEGER` Signed-off-by: Miki <miki@amazon.com>
1ac106c
to
d697969
Compare
….MAX_SAFE_INTEGER` (opensearch-project#544) Signed-off-by: Miki <miki@amazon.com>
* Manually sync CHANGELOG.md, release-drafter.yml, RELEASING.md, and USER_GUIDE.md * Version Bump: 2.3.0 (opensearch-project#546) * Upgrade and secure the backport workflow (opensearch-project#547) * Add serialization and deserialization of numerals larger than `Number.MAX_SAFE_INTEGER` (opensearch-project#544) * Add upgrading NPM to all workflows running older Node.js versions (opensearch-project#545) * Bump @types/node from 20.1.4 to 20.2.5 (opensearch-project#528) * Bump @aws-sdk/types from 3.329.0 to 3.341.0 (opensearch-project#527) * Bump ora from 6.3.0 to 6.3.1 (opensearch-project#524) * Bump node-fetch from 3.2.10 to 3.3.1 (opensearch-project#526) * Bump eslint from 8.39.0 to 8.41.0 (opensearch-project#525) * Bump @types/node from 18.16.0 to 20.1.4 (opensearch-project#518) * Bump xmlbuilder2 from 3.1.0 to 3.1.1 (opensearch-project#519) * Bump semver from 7.3.8 to 7.5.1 (opensearch-project#520) * Added timursaurus to MAINTAINERS.md (opensearch-project#517) * Bump @aws-sdk/types from 3.257.0 to 3.329.0 (opensearch-project#516) * Bump @babel/eslint-parser from 7.21.3 to 7.21.8 (opensearch-project#513) * Bump simple-git from 3.17.0 to 3.18.0 (opensearch-project#512) * Bump xmlbuilder2 from 3.0.2 to 3.1.0 (opensearch-project#503) * Bump eslint from 8.38.0 to 8.39.0 (opensearch-project#504) * Bump @types/node from 18.15.11 to 18.16.0 (opensearch-project#499) * Bump ora from 6.1.2 to 6.3.0 (opensearch-project#500) * Bump prettier from 2.8.7 to 2.8.8 (opensearch-project#501) * Implemented Docker Image caching for `integration-unreleased` workflow (opensearch-project#498) * [Bug] Fixed bundler.yml (opensearch-project#497) * Bumped Version 2.2.1 In preparation for release (opensearch-project#495) * Bump @types/node from 18.15.10 to 18.15.11 (opensearch-project#493) * Bump eslint from 8.33.0 to 8.38.0 (opensearch-project#494) * Bump rimraf from 4.4.0 to 5.0.0 (opensearch-project#492) * [CCI] feat: add missing createConnection type (opensearch-project#490) * Add Search guide (opensearch-project#489) * Create search guide (opensearch-project#488) * [CCI] Create advanced index actions guide (opensearch-project#483) * [CCI] Connection `request` method callback (opensearch-project#478) * feat: NotCompatibleError (opensearch-project#484) * Bump @babel/eslint-parser from 7.19.1 to 7.21.3 (opensearch-project#486) * Bump jsdoc from 4.0.0 to 4.0.2 (opensearch-project#485) * Bump prettier from 2.8.4 to 2.8.7 (opensearch-project#487) * Create document lifecycle guide (opensearch-project#481) * Create index lifecycle guide (opensearch-project#482) * Create bulk guide (opensearch-project#480) * [CCI] Create index_template guide (opensearch-project#479) * Bump split2 from 4.1.0 to 4.2.0 (opensearch-project#467) * Bump deepmerge from 4.3.0 to 4.3.1 (opensearch-project#468) * Bump @types/node from 18.15.3 to 18.15.10 (opensearch-project#462) * Bump eslint-config-prettier from 8.6.0 to 8.8.0 (opensearch-project#463) * [CCI] Update developerGuide regarding yarn troubleshoot steps (opensearch-project#456) * [CCI] Bump caniuse-lite from 1.0.30001249 to 1.0.30001469 (opensearch-project#459) * added the solution for the possible error during yarn installation on Windows OS (opensearch-project#453) * Fixed deprecation warnings (opensearch-project#446) * [CCI] Don't run tests on PRs with doc changes only (opensearch-project#441) * [CCI] Add pull request template (opensearch-project#440) * Bump @types/node from 17.0.45 to 18.15.3 (opensearch-project#448) * Bump simple-git from 3.16.0 to 3.17.0 (opensearch-project#447) * [CCI] Update Developer Guide (opensearch-project#436) * [CCI] Remove waitCluster in Integration Tests (opensearch-project#423) * Bump tsd from 0.25.0 to 0.27.0 (opensearch-project#431) * Bump rimraf from 4.1.1 to 4.4.0 (opensearch-project#432) * [CCI] doc: fix grammar (opensearch-project#427) * Fix deprecated folder mapping "./" in the "exports" field (opensearch-project#416) * [CCI] fix: pass required `data` argument to SerializationError class (opensearch-project#419) * Bump tap from 16.3.0 to 16.3.4 (opensearch-project#413) * [CCI] Refactor: Remove unnecessary `data` argument when invoking `OpenSearchClientError` (opensearch-project#421) * Bump deepmerge from 4.2.2 to 4.3.0 (opensearch-project#414) * Updated Maintainers list and changelogs (opensearch-project#409) * Downgraded @types/node to 17 from 18 due to breaking changes. (opensearch-project#405) * Bump aws4 from 1.11.0 to 1.12.0 (opensearch-project#406) * Bump prettier from 2.8.3 to 2.8.4 (opensearch-project#407) * Bump minimist from 1.2.7 to 1.2.8 (opensearch-project#388) * Bump tsd from 0.24.1 to 0.25.0 (opensearch-project#389) * Add missing types for AwsSigv4SignerOptions.service (opensearch-project#377) * Created untriaged issue workflow. (opensearch-project#386) Daniel (dB.) Doubrovkine* 2/14/23, 12:14 PM * Bump dezalgo from 1.0.3 to 1.0.4 (opensearch-project#383) * Bump simple-statistics from 7.7.0 to 7.8.3 (opensearch-project#384) * Bump eslint from 8.32.0 to 8.33.0 (opensearch-project#379) * Allow fields in BulkOperation to be optional (opensearch-project#378) * Bump @types/node from 18.11.18 to 18.11.19 (opensearch-project#380) * Bump ora from 5.4.1 to 6.1.2 (opensearch-project#376) * Bump @aws-sdk/types from 3.226.0 to 3.257.0 (opensearch-project#375) * Bump secure-json-parse from 2.4.0 to 2.7.0 (opensearch-project#369) * Bump simple-git from 3.15.1 to 3.16.0 (opensearch-project#373) * Updated user guide to include Amazon OpenSearch Serverless (opensearch-project#372) * Added Support for AOSS (opensearch-project#366) * Bump rimraf from 3.0.2 to 4.1.1 (opensearch-project#370) * Bump eslint-config-prettier from 8.5.0 to 8.6.0 (opensearch-project#368) * Bump hpagent from 0.1.2 to 1.2.0 (opensearch-project#361) * Bump eslint from 8.30.0 to 8.32.0 (opensearch-project#362) * Fixed missing namespace for API-PIT endpoints (opensearch-project#364) * Bump prettier from 2.7.1 to 2.8.3 (opensearch-project#363) * Ensure Dependabot PR workflow retriggers on label change (opensearch-project#360) * Added point in time APIs (opensearch-project#348) * Bump @types/node from 15.14.7 to 18.11.18 (opensearch-project#353) * Bump split2 from 3.2.2 to 4.1.0 (opensearch-project#354) * Bump json5 from 2.2.0 to 2.2.3 (opensearch-project#359) * Bump @aws-sdk/types from 3.190.0 to 3.226.0 (opensearch-project#355) * Updated MAINTAINERS.md to match recommended opensearch-project format. (opensearch-project#358) Daniel (dB.) Doubrovkine* 1/6/23, 8:36 AM * Bump minimist from 1.2.6 to 1.2.7 (opensearch-project#350) * Bump eslint from 7.32.0 to 8.30.0 (opensearch-project#346) * Bump eslint-plugin-prettier from 4.0.0 to 4.2.1 (opensearch-project#351) * Removed test artifacts from gh_pages workflow (opensearch-project#347) * Bump minimatch from 3.0.4 to 3.1.2 (opensearch-project#345) * Bump xmlbuilder2 from 2.4.1 to 3.0.2 (opensearch-project#321) * Implement JSDOC (opensearch-project#337) * Added skip-changelog label (opensearch-project#339) * Bumps simple-git from 3.4.0 to 3.15.0 (opensearch-project#341) * [FEATURE] Allow overriding the aws service identifier in AwsSigv4Signer (opensearch-project#333) * Add release details and bump the jenkins lib version (opensearch-project#319) * Bump Version from 2.1.0 to 2.1.1 (opensearch-project#318) * Bump semver from 7.3.7 to 7.3.8 (opensearch-project#313) * Add release workflows (opensearch-project#317) * issue opensearch-project#291 mutability of headers (opensearch-project#311) * Bump tsd from 0.22.0 to 0.24.1 (opensearch-project#312) * Add changelog and changelog verifier (opensearch-project#306) * Bump prettier from 2.6.2 to 2.7.1 (opensearch-project#310) * Bump @aws-sdk/types from 3.160.0 to 3.190.0 (opensearch-project#309) * Bump node-fetch from 2.6.7 to 3.2.10 (opensearch-project#281) * Update Maintainers List (opensearch-project#308) * Implemented Retry for OpenSearch Container (opensearch-project#304) * Feature/add default credentials provider (opensearch-project#295) * Fix new line problem between diffs (opensearch-project#302) * Removed Unused Variables (opensearch-project#301) * fix awssigv4signer.test.js tests not running (opensearch-project#294) * fix: support TS resolution for .mjs (opensearch-project#296) * Add AwsSigV4 signing functionality (opensearch-project#279) * Fix opensearch-project#253 Cannot read property 'then' of null in Transport.js issue. (opensearch-project#283) * Bump simple-git from 3.5.0 to 3.13.0 (opensearch-project#286) * Bump semver from 7.3.5 to 7.3.7 (opensearch-project#280) * Adds bulk example to README (opensearch-project#277) * Adding Dependabot configuration (opensearch-project#278) * Removing OpenDistro integration tests (opensearch-project#271) * Upgrade dependencies (opensearch-project#272) * fix: add missing memory circuit breaker options (opensearch-project#266) * feat: allow `doc` overwrite in `onDocument` (opensearch-project#263) * Adding link checker workflow (opensearch-project#262) * Using standardized templates from .github (opensearch-project#249) * Adding new OpenSearch versions and updating compatibility matrix (opensearch-project#257) Signed-off-by: Miki <miki@amazon.com>
….MAX_SAFE_INTEGER` (opensearch-project#544) Signed-off-by: Miki <miki@amazon.com>
….MAX_SAFE_INTEGER` (opensearch-project#544) Signed-off-by: Miki <miki@amazon.com>
* Add upgrading NPM to all workflows running older Node.js versions (#545) (#553) Also: * Add compatibility checks for Node.js v18 * Bump `actions/setup-node` to v3 Signed-off-by: Miki <miki@amazon.com> * Add serialization and deserialization of numerals larger than `Number.MAX_SAFE_INTEGER` (#544) (#554) Signed-off-by: Miki <miki@amazon.com> * Version Bump: 2.3.0 (#546) (#555) Signed-off-by: Theo Truong <theotr@amazon.com> * Make handling of long numerals an option that is disabled by default (#557) (#561) Also: * Strengthen the tests * update USER_GUIDE.md (cherry picked from commit 08069bc) Signed-off-by: Miki <miki@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Version Bump: 2.3.1 Signed-off-by: Theo Truong <theotr@amazon.com> --------- Signed-off-by: Miki <miki@amazon.com> Signed-off-by: Theo Truong <theotr@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Miki <miki@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Description
In JavaScript, a
Number
is a 64-bit floating-point value which can store 16-ish digits. However, the serializer and deserializer will need to cater to numeric values generated by other languages which can have up to 19 digits. Native JSON parser and stringifier, incapable of handling the extra digits, corrupt the values, making them unusable.To work around this limitation, with this change, the deserializer converts long sequences of digits into strings and marks them before applying the parser. During the parsing, string values that begin with the mark are converted to
BigInt
values. Similarly, during stringification, the serializer convertsBigInt
values to marked strings and when done, it replaces them with plain numerals.Issues Resolved
Related to opensearch-project/OpenSearch-Dashboards#4536
Check List
yarn run lint
doesn't show any errorsBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.