Missing toByteArray

[cyberphone]

A very common representation of BigInt in JSON is using Base64Url coded strings. Although doable, the solution offered by for example Java is way easier:
https://docs.oracle.com/javase/8/docs/api/java/math/BigInteger.html#toByteArray--

This is the pretty ugly and probably buggy workaround I ended up with:

// Browser specific solution
BigInt.prototype.toJSON = function() {
  hex2bin = function(c) {
    return c - (c < 58 ? 48 : 87);
  };
  let v = this.valueOf();
  let sign = false;
  if (v < 0) {
    v = -v;
    sign = true;
  }
  let hex = v.toString(16);
  if (hex.length & 1) hex = '0' + hex;
  let binary = new Uint8Array(hex.length / 2);
  let i = binary.length;
  let q = hex.length;
  let carry = 1;
  while(q > 0) {
     let byte = hex2bin(hex.charCodeAt(--q)) + (hex2bin(hex.charCodeAt(--q)) << 4);
     if (sign) {
       byte = ~byte + carry;
       if (byte > 255) {
         carry = 1;
       } else {
         carry = 0;
       }
     }
     binary[--i] = byte;
  }
  if (!sign && binary[0] > 127) {
    let binp1 = new Uint8Array(binary.length + 1);
    binp1[0] = 0;
    for (q = 0; q < binary.length; q++) {
      binp1[q + 1] = binary[q];
    }
    binary = binp1;
  }
  let text = '';
  for (q = 0; q < binary.length; q++) {
    text += String.fromCharCode(binary[q]);
  }
  return window.btoa(text)
    .replace(/\+/g,'-').replace(/\//g,'_').replace(/=/g,'');
}

JSON.stringify({big: 555555555555555555555555555555n, small:55});

Expected result: '{"big":"BwMYyOV8edmCI4444w","small":55}'

Also see: #162 (comment)

[jakobkummerow]

The BigInt proposal intentionally leaves extended functionality for future proposals (see https://github.com/tc39/proposal-bigint#left-for-future-proposals ). This sounds like a candidate for a possible future BigInt library proposal. (I don't know if or where any work has started yet on this.)

[littledan]

As @jakobkummerow says, the library here is intentionally minimal. We don't have a particular place where we're collecting library proposals. For now, we can use this bug tracker. After the core of BigInt gets to Stage 4, we let's start getting more concrete about what we want to do here for a BigInt standards library.

[no2chem]

+1 on this.

For node, I wrote https://github.com/no2chem/bigint-buffer --- it works in the browser too, relying on Buffer.from(val, 'hex') and the polyfill provided by webpack.

But yeah, getting real use of BigInts is going to need conversion to/from UInt8Arrays. I'm surprised the proposal made it into stage 3 without it, because it seems like this conversions form a typical use case of BigInts.

[littledan]

@no2chem It's great that you were able to do this kind of experimental development of libraries with BigInt. Let's continue developing in this mode for a while, and then consider what we want to standardize.

[MicahZoltu]

@jakobkummerow There is no longer a section for left-for-future-proposals in the linked document. Has it been moved?

When this gets considered in the future, please ensure that both signed and unsigned are supported. For example, at the moment, it appears that I can do BigInt(new Uint8Array([0x20])) to get an unsigned BigInt from a Uint8Array, but I don't see any way to interpret the Uint8Array as a twos-complement signed number. This means that while I can deserialize an unsigned BigInt, I cannot deserialize a signed BigInt. It appears that BigInt(new Uint8Array([0x20])) is actually just JavaScript being horrible and implicitly converting my 1-element array into a number, which it then passes to BigInt. I would like to avoid more such mistakes in the future.

[MicahZoltu]

For now, these functions (TypeScript) should allow you to convert from Uint8Array to bigint and back using two's complement for signed numbers. If anyone is any good at mathing, please double-check my work!

Consider this Unlicensed/CC0/public domain/MIT/Apache 2.0/FreeBSD (you choose) licensed for the purpose of using it wherever you want (I truly don't care what you do with it or whether you give any credit). These functions are written for readability, not for performance. There are definitely opportunities for optimizations if you feel like it.

export function uint8ArrayToBigint(uint8Array: Uint8Array, numberOfBits: number): bigint {
	if (numberOfBits % 8) throw new Error(`Only 8-bit increments are supported when (de)serializing a bigint.`)
	const valueAsHexString = uint8ArrayToHexString(uint8Array)
	return hexStringToBigint(valueAsHexString, numberOfBits)
}

export function hexStringToBigint(hexString: string, numberOfBits: number): bigint {
	if (numberOfBits % 8) throw new Error(`Only 8-bit increments are supported when (de)serializing a bigint.`)
	const unsignedInterpretation = BigInt(validateAndNormalizeHexString(hexString))
	return twosComplement(unsignedInterpretation, numberOfBits)
}

export function bigintToUint8Array(value: bigint, numberOfBits: number): Uint8Array {
	if (numberOfBits % 8) throw new Error(`Only 8-bit increments are supported when (de)serializing a bigint.`)
	const valueAsHexString = bigintToHexString(value, numberOfBits)
	return hexStringToUint8Array(valueAsHexString)
}

export function bigintToHexString(value: bigint, numberOfBits: number): string {
	if (numberOfBits % 8) throw new Error(`Only 8-bit increments are supported when (de)serializing a bigint.`)
	const valueToSerialize = twosComplement(value, numberOfBits)
	return unsignedBigintToHexString(valueToSerialize, numberOfBits)
}

function validateAndNormalizeHexString(hex: string): string {
	const match = new RegExp(`^(?:0x)?([a-fA-F0-9]*)$`).exec(hex)
	if (match === null) throw new Error(`Expected a hex string encoded byte array with an optional '0x' prefix but received ${hex}`)
	if (match.length % 2) throw new Error(`Hex string encoded byte array must be an even number of charcaters long.`)
	return `0x${match[1]}`
}

function uint8ArrayToHexString(array: Uint8Array): string {
	const hexStringFromByte = (byte: number): string => ('00' + byte.toString(16)).slice(-2)
	const appendByteToString = (value: string, byte: number) => value + hexStringFromByte(byte)
	return array.reduce(appendByteToString, '')
}

function hexStringToUint8Array(hex: string): Uint8Array {
	const match = new RegExp(`^(?:0x)?([a-fA-F0-9]*)$`).exec(hex)
	if (match === null) throw new Error(`Expected a hex string encoded byte array with an optional '0x' prefix but received ${hex}`)
	if (match.length % 2) throw new Error(`Hex string encoded byte array must be an even number of charcaters long.`)
	const normalized = match[1]
	const byteLength = normalized.length / 2
	const bytes = new Uint8Array(byteLength)
	for (let i = 0; i < byteLength; ++i) {
		bytes[i] = (Number.parseInt(`${normalized[i*2]}${normalized[i*2+1]}`, 16))
	}
	return bytes
}

function unsignedBigintToHexString(value: bigint, bits: number): string {
	const byteSize = bits / 8
	const hexStringLength = byteSize * 2
	return ('0'.repeat(hexStringLength) + value.toString(16)).slice(-hexStringLength)
}

function twosComplement(value: bigint, numberOfBits: number): bigint {
	const mask = 2n**(BigInt(numberOfBits) - 1n) - 1n
	return (value & mask) - (value & ~mask)
}

And here are some tests to show that my functions and tests have the same bugs.

import { expect } from 'chai'
import { hexStringToBigint, bigintToHexString, uint8ArrayToBigint, bigintToUint8Array } from './index'

const testCases: Array<[bigint, string | Uint8Array]> = [
	[0n, '0000000000000000000000000000000000000000000000000000000000000000'],
	[1n, '0000000000000000000000000000000000000000000000000000000000000001'],
	[2n, '0000000000000000000000000000000000000000000000000000000000000002'],
	[57896044618658097711785492504343953926634992332820282019728792003956564819966n, '7ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe'],
	[57896044618658097711785492504343953926634992332820282019728792003956564819967n, '7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff'],
	[-57896044618658097711785492504343953926634992332820282019728792003956564819968n, '8000000000000000000000000000000000000000000000000000000000000000'],
	[-57896044618658097711785492504343953926634992332820282019728792003956564819967n, '8000000000000000000000000000000000000000000000000000000000000001'],
	[-57896044618658097711785492504343953926634992332820282019728792003956564819966n, '8000000000000000000000000000000000000000000000000000000000000002'],
	[-2n, 'fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe'],
	[-1n, 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff'],
	[0n, '00'],
	[1n, '01'],
	[2n, '02'],
	[126n, '7e'],
	[127n, '7f'],
	[-128n, '80'],
	[-127n, '81'],
	[-126n, '82'],
	[-2n, 'fe'],
	[-1n, 'ff'],
	[0n, new Uint8Array([0x00])],
	[1n, new Uint8Array([0x01])],
	[2n, new Uint8Array([0x02])],
	[126n, new Uint8Array([0x7e])],
	[127n, new Uint8Array([0x7f])],
	[-128n, new Uint8Array([0x80])],
	[-127n, new Uint8Array([0x81])],
	[-126n, new Uint8Array([0x82])],
	[-2n, new Uint8Array([0xfe])],
	[-1n, new Uint8Array([0xff])],
]

for (let testCase of testCases) {
	const expected = testCase[0]
	const size = testCase[1].length * ((typeof testCase[1] === 'string') ? 4 : 8)
	const actual = (typeof testCase[1] === 'string') ? hexStringToBigint(testCase[1], size) : uint8ArrayToBigint(testCase[1], size)
	expect(actual).to.equal(expected)
}

for (let testCase of testCases) {
	const expected = (typeof testCase[1] === 'string') ? testCase[1] : testCase[1].toString()
	const size = testCase[1].length * ((typeof testCase[1] === 'string') ? 4 : 8)
	const actual = (typeof testCase[1] === 'string') ? bigintToHexString(testCase[0], size) : bigintToUint8Array(testCase[0], size).toString()
	expect(actual).to.equal(expected)
}

[MicahZoltu]

@caiolima Why was this closed? #163 (comment) says:

We don't have a particular place where we're collecting library proposals. For now, we can use this bug tracker. After the core of BigInt gets to Stage 4, we let's start getting more concrete about what we want to do here for a BigInt standards library.

[caiolima]

@MicahZoltu This proposal is already stage 4 and merged and I'm doing housekeeping of the repository. This repository is going to be archived soon and I'm afraid we won't be able to keep the discussion here, since archived repositories are read-only. AFICT, discussion of future proposals before stage 1 are happening in https://es.discourse.group. This thread is going to be available to be referenced, but it won't be possible to post new comments.

[MicahZoltu]

Created a proposal over there for anyone who comes along later: https://es.discourse.group/t/bigint-enhancements/100