Use BigInt over Number for int64 #1557

mjgp2 · 2021-03-09T13:47:46Z

This PR removes the need for the Long package, and instead of Number uses BigInt as standard for all int64 types.

I realise that this would be a breaking change, but it seems like it would be the right design decision for the next major version of this library?

All unit tests pass, and needed to switch to terser from uglify.

Node versions before 10 would not be supported. However, Node 10 LTS is EOL on 2021-04-30.

alexander-fenster · 2021-03-12T23:49:51Z

Hi @mjgp2,

Thank you for this PR. We probably cannot include it in v7.0.0 (that is coming soon) but we'll think if we can drop long in the next semver major. At least this would be a major pain for us at Google to convert all the client libraries (which reexport types from protobuf.js), and I can only imagine how disruptive it can be for others. Also, if you see how the long support is implemented, there's a configuration option that enables or disables long. It maybe makes sense to think if we can make this less breaking by doing it in steps:

(1) start accepting BigInt where we currently accept long when serializing;
(2) make it configurable to be able to choose if we dump long or BigInt.

Also, dropping support for Node.js v4-6-8 is a big decision.

Of course, Cc: @dcodeIO on this.

I promise we'll get back to reviewing this PR when we're done with releasing v7.0.0 :)

mjgp2 · 2021-03-15T10:09:57Z

Thanks for your reply.

Yes, I think that definitely you could support Number, BigInt, String and Long for int64 types. My thinking here was firstly to understand the codebase and to swap Long for BigInt being simpler than adding the extra branching logic, and then hopefully get some feedback.

I would suggest that it is considered whether support for Number for int64 should be dropped - there are silent overflows today I think within the codebase that are really data corruption.

ardatan · 2021-04-20T08:39:13Z

Any progress on this PR?

mjgp2 · 2021-06-25T16:19:21Z

@ardatan - you can use the branch on my repo, it works, i.e. in your package.json

  "resolutions": {
    "**/protobufjs": "mjgp2/protobuf.js#bigint"
  },

I understand it's going to be targeted for version 8. I'm happy to help when the time comes - it's still my thinking that Number support for int64 should be dropped in version 8 because it will corrupt data.

hornta · 2021-09-17T16:38:46Z

Sorry for pinging a bunch of people but me and my colleagues are really looking forward to this PR being merged and published so we are looking for a status on this. Is it stuck on something other than having conflicts? Can we help out in any way?

mjgp2 · 2021-09-17T17:54:30Z

I'm happy to take it forward, but I think I was waiting for v7 to be released first. Not sure where that is right now?

mouafa · 2021-12-01T12:23:43Z

Well done @mjgp2

Is there some sort of a timeline with a release date for v7 and v8?, I'm really looking forward to dropping the ~40KB long.js from our codebase.

goya · 2021-12-01T15:59:38Z

just use this https://gist.github.com/goya/8185996e60725371bb30971bd641aad4

bbeausej · 2022-01-20T18:06:51Z

tests/data/convert.js

-            } else
-                object.uint64Val = options.longs === String ? "0" : 0;
+            var long = new $util.LongBits(NaN, NaN, undefined);
+            object.uint64Val = options.longs === String ? long.toBigInt().toString() : options.longs === BigInt ? long.toBigInt().toString() : long;


Potential bug fix to default value here if the user chooses to use { longs: BitInt }

Suggested change

object.uint64Val = options.longs === String ? long.toBigInt().toString() : options.longs === BigInt ? long.toBigInt().toString() : long;

object.uint64Val = options.longs === String ? long.toBigInt(true).toString() : options.longs === BigInt ? long.toBigInt(true): long;

mjgp2 · 2022-01-25T15:49:22Z

@alexander-fenster - wondering if there is an ETA for v7 so I can start to work out how to take this forward?

alexander-fenster · 2022-02-02T01:33:10Z

@mjgp2 Sorry, I don't have access to releases in this repository so I cannot help here. Your best bet would be to ask the owner, @dcodeIO.

achingbrain · 2022-03-30T12:36:49Z

src/util/longbits.js

- * Zero hash.
- * @memberof util.LongBits
- * @type {string}
+ * Constructs new long bits from the specified number.


Suggested change

* Constructs new long bits from the specified number.

* Constructs new long bits from the specified bigint.

achingbrain · 2022-03-30T12:36:52Z

src/util/longbits.js

- * @memberof util.LongBits
- * @type {string}
+ * Constructs new long bits from the specified number.
+ * @param {number} value Value


Suggested change

* @param {number} value Value

* @param {bigint} value Value

achingbrain · 2022-03-30T12:36:53Z

src/util/longbits.js

 */
-var zeroHash = LongBits.zeroHash = "\0\0\0\0\0\0\0\0";
+LongBits.fromBigInt = function fromNumber(value) {


Suggested change

LongBits.fromBigInt = function fromNumber(value) {

LongBits.fromBigInt = function fromBigInt(value) {

mahnunchik · 2022-07-19T14:13:56Z

Any news?

mjgp2 · 2022-07-19T15:03:33Z

Well, v7 has been released (woo!), and so hopefully we can start to pick this up @dcodeIO ?

alexander-fenster · 2022-07-20T08:12:53Z

Yes, we finally got v7 out - and also dropped support for older versions of Node.js (we start from 12 now). I should say I'm not sure how we properly release this if we make this change, this will be a huge breaking change for many users. I wonder if we could somehow use the configuration option to choose which type to use (and have this option for pbts as well, since the generated types are also affected).

achingbrain · 2022-08-08T19:54:09Z

src/util/longbits.js

+    var hi = Number(value >> 32n) | 0;
+    var lo = Number(value - ( BigInt(hi) << 32n ) ) | 0;


I've been unable to correctly encode uint64 fields with the max field value of 18446744073709551615n without this change:

Suggested change

var hi = Number(value >> 32n) | 0;

var lo = Number(value - ( BigInt(hi) << 32n ) ) | 0;

var hi = Number(value >> 32n);

var lo = Number(value - ( BigInt(hi) << 32n ) );

I fixed it by juzibot#7

Ports just the parts of protobufjs we use to typescript and integrates the changes from protobufjs/protobuf.js#1557 to have native BigInt support.

hallahan · 2022-09-02T03:47:34Z

I'm assuming bigint will perform significantly better since it's a native type. If we aren't going to merge this, would it make sense to fork this repo and have an npm package under a different name?

Having an object wrapping two Numbers uses significantly more memory as well, so wouldn't it be advantageous to remove Long? We could add a Long shim around bigint for backwards compatibility.

mahnunchik · 2022-10-26T12:33:40Z

Any news?

Maligosus · 2022-11-08T04:45:21Z

Any news?

azizghuloum · 2022-12-28T04:31:29Z

I'm not a maintainer so speaking for myself here.

Honestly I don't think this can be merged due to breaking compatibility.

What needs to be done is:

First: Add support for bigints as an option (so people can migrate to it).

Then: Maybe in 5 years, drop support of Long (since no one would be using it by then, hopefully).

It's a great patch and I wish it were merged already. I need bigints myself and don't want to start a new project with legacy junk from the start. However, I can imagine that there are millions of lines of code affected by this change and that's why it's been 2 years since the patch was written and it's still not merged. It breaks people's code.

Yes, the maintainers can merge it and force people to migrate or stay behind. However, people will generally not like that.

@dcodeIO @alexander-fenster @mjgp2 Is this kinda what the situation is? Would this be a reasonable approach to make progress?

hallahan · 2022-12-28T05:45:52Z

Why not just merge this and release a full version bump? That's what versions are for. Software that wants to keep using Long can stick with an older version.

Putting two Numbers (which are float64) in an Object to handle integers over 52 bits is a very inefficient solution.

goya · 2022-12-28T07:36:20Z

i agree merge this. i vote we merge this 1 year after bigint has been released in stable JS versions.

“millions of lines of code” = code generated by this library.

mjgp2 · 2022-12-31T07:07:11Z

i agree merge this. i vote we merge this 1 year after bigint has been released in stable JS versions.

Bigint is in all major browsers for over 2 years and since Node 10.

I think it's waiting for the owner(s) to decide how to proceed - there is a decision to be made about if it is necessary to support both Long and BigInt. I do not generate code with this library, just use it dynamically, so do not have much context on how it affects that.

mahnunchik · 2023-05-24T11:42:18Z

@alexander-fenster ping from 2023

phoenix741 · 2023-05-24T12:27:03Z

I wait for native bigint too

seo-rii · 2023-06-20T05:01:38Z

src/converter.js

-            ("d%s=o.longs===String?n.toString():o.longs===Number?n.toNumber():n", prop)
-        ("}else")
-            ("d%s=o.longs===String?%j:%i", prop, field.typeDefault.toString(), field.typeDefault.toNumber());
+            ("var n=new util.LongBits(%i,%i,%j)", field.typeDefault.low, field.typeDefault.high, field.typeDefault.unsigned)


This code will define n for every long field, which cause failure on es6 build(-w es6).
You should check if n has declared in previous field.
Like this : seo-rii@1dac06f#diff-444e086c8038faf674140e30c0597966b9a9102859bf04422d247118adedd08c

jtbandes · 2023-07-06T00:35:17Z

I created a PR that's smaller in scope: #1908 which adds support for toObject({ longs: BigInt }).

Ports just the parts of protobufjs we use to typescript and integrates the changes from protobufjs/protobuf.js#1557 to have native BigInt support.

jmondo · 2023-12-20T17:21:31Z

the concerns about breaking changes seem like they are complicating this PR, but could we get some 👀 on @jtbandes's PR #1908 ? it seems like a simple first step that would at least unblock users like myself who want to opt in to using BigInt 😄

shanghaikid · 2024-02-21T08:52:52Z

2024, any progress?

mjgp2 · 2024-02-21T08:57:31Z

I don't have bandwidth to look at this any further - hopefully someone will be able to get some form of support for BigInt in.

hallahan · 2024-02-21T14:25:53Z

😢

mahnunchik · 2024-02-21T17:46:07Z

😢 😢 😢

cupen · 2024-03-21T10:19:36Z

Lol.
It seems that int64/uint64 is a big problem to javascript.

This was referenced Mar 9, 2021

add force-bigint arg #1314

Open

BigInt in node.js #1151

Open

Support bigint by default

a04c0e3

mjgp2 force-pushed the bigint branch from bece9d1 to a04c0e3 Compare March 9, 2021 14:05

Switch to terser

ee2f583

mjgp2 force-pushed the bigint branch from 41ac00a to ee2f583 Compare March 9, 2021 14:57

Drop node < 10 support

e24adf2

mjgp2 force-pushed the bigint branch from 653ed16 to e24adf2 Compare March 10, 2021 17:08

alexander-fenster added the 8.x label Mar 12, 2021

cometkim mentioned this pull request Apr 6, 2021

grpc handler does not work with int64 type record ids ardatan/graphql-mesh#1355

Closed

ardatan mentioned this pull request Apr 20, 2021

Support bigint as input #1592

Open

ermyas mentioned this pull request May 11, 2021

Fabric Interop Node SDK Support for HTLC-based Asset Exchanges hyperledger-labs/weaver-dlt-interoperability#45

Merged

altro3 mentioned this pull request May 29, 2021

Option to use bigint primitive or BigInt wrapper instead of long.js stephenh/ts-proto#296

Open

achingbrain mentioned this pull request Jun 6, 2021

feat: validate v2 ipns signatures ipfs/js-ipns#121

Merged

DonghunLouisLee mentioned this pull request Aug 1, 2021

Large int64 values are truncated spluxx/Protoman#39

Closed

bbeausej reviewed Jan 24, 2022

View reviewed changes

achingbrain reviewed Mar 30, 2022

View reviewed changes

dyladan mentioned this pull request Aug 4, 2022

ValueType.INT metric instrument integer overflow open-telemetry/opentelemetry-js#3014

Open

achingbrain reviewed Aug 8, 2022

View reviewed changes

achingbrain mentioned this pull request Aug 9, 2022

fix!: increase encoding/decoding performance ipfs/protons#58

Merged

achingbrain added a commit to ipfs/protons that referenced this pull request Aug 10, 2022

fix: port protobuf reader/writer to ts

e8c165a

Ports just the parts of protobufjs we use to typescript and integrates the changes from protobufjs/protobuf.js#1557 to have native BigInt support.

achingbrain mentioned this pull request Aug 10, 2022

fix: port protobuf reader/writer to ts ipfs/protons#60

Merged

binsee mentioned this pull request Sep 22, 2022

feat: support bigint juzibot/protobuf.js#1

Merged

azizghuloum mentioned this pull request Dec 28, 2022

ts-proto 2.x ideas stephenh/ts-proto#647

Open

dk1a mentioned this pull request Feb 19, 2023

refactor(cli): refactor to esm latticexyz/mud#417

Merged

seo-rii suggested changes Jun 20, 2023

View reviewed changes

jtbandes mentioned this pull request Jul 6, 2023

Add support for longs: BigInt in toObject #1908

Open

achingbrain added a commit to ipfs/protons that referenced this pull request Oct 13, 2023

fix: port protobuf reader/writer to ts (#60)

d101804

Ports just the parts of protobufjs we use to typescript and integrates the changes from protobufjs/protobuf.js#1557 to have native BigInt support.

mjgp2 closed this Feb 21, 2024

	object.uint64Val = options.longs === String ? long.toBigInt().toString() : options.longs === BigInt ? long.toBigInt().toString() : long;
	object.uint64Val = options.longs === String ? long.toBigInt(true).toString() : options.longs === BigInt ? long.toBigInt(true): long;

	* Constructs new long bits from the specified number.
	* Constructs new long bits from the specified bigint.

	LongBits.fromBigInt = function fromNumber(value) {
	LongBits.fromBigInt = function fromBigInt(value) {

		var hi = Number(value >> 32n) \| 0;
		var lo = Number(value - ( BigInt(hi) << 32n ) ) \| 0;

Use BigInt over Number for int64 #1557

Use BigInt over Number for int64 #1557

Conversation

mjgp2 commented Mar 9, 2021 • edited Loading

alexander-fenster commented Mar 12, 2021

mjgp2 commented Mar 15, 2021

ardatan commented Apr 20, 2021

mjgp2 commented Jun 25, 2021

hornta commented Sep 17, 2021 • edited Loading

mjgp2 commented Sep 17, 2021

mouafa commented Dec 1, 2021

goya commented Dec 1, 2021

bbeausej Jan 20, 2022

Choose a reason for hiding this comment

mjgp2 commented Jan 25, 2022

alexander-fenster commented Feb 2, 2022

achingbrain Mar 30, 2022

Choose a reason for hiding this comment

achingbrain Mar 30, 2022

Choose a reason for hiding this comment

achingbrain Mar 30, 2022

Choose a reason for hiding this comment

mahnunchik commented Jul 19, 2022

mjgp2 commented Jul 19, 2022

alexander-fenster commented Jul 20, 2022

achingbrain Aug 8, 2022

Choose a reason for hiding this comment

binsee Dec 7, 2022

Choose a reason for hiding this comment

hallahan commented Sep 2, 2022

mahnunchik commented Oct 26, 2022

Maligosus commented Nov 8, 2022

azizghuloum commented Dec 28, 2022 • edited Loading

hallahan commented Dec 28, 2022

goya commented Dec 28, 2022

mjgp2 commented Dec 31, 2022

mahnunchik commented May 24, 2023

phoenix741 commented May 24, 2023

seo-rii Jun 20, 2023

Choose a reason for hiding this comment

jtbandes commented Jul 6, 2023

jmondo commented Dec 20, 2023

shanghaikid commented Feb 21, 2024

mjgp2 commented Feb 21, 2024

hallahan commented Feb 21, 2024

mahnunchik commented Feb 21, 2024

cupen commented Mar 21, 2024 • edited Loading

mjgp2 commented Mar 9, 2021 •

edited

Loading

hornta commented Sep 17, 2021 •

edited

Loading

azizghuloum commented Dec 28, 2022 •

edited

Loading

cupen commented Mar 21, 2024 •

edited

Loading