Buffer created using new Buffer(buffer) has a different ArrayBuffer from the original one

[ghost]

Hope the title describes the issue correctly:)

Codes:

var typed_array = new Uint8Array(6);
for(var i = 0; i < typed_array.length; i++){
  typed_array[i] = i;
}

var array_buffer = typed_array.buffer;
console.log("byteLength of ArrayBuffer : " + array_buffer.byteLength);

var node_buffer = new Buffer(array_buffer);
console.log("Node Buffer : ");
console.log(node_buffer);

console.log("byteLength of ArrayBuffer in Node Buffer : ", node_buffer.buffer.byteLength);

var another_node_buffer = new Buffer(node_buffer);
console.log("Another Node Buffer : ");
console.log(another_node_buffer);

console.log("byteLength of ArrayBuffer in Another Node Buffer : ", another_node_buffer.buffer.byteLength);

node_buffer.buffer is different from another_node_buffer.buffer !

[ChALkeR]

This is expected behaviour.

var node_buffer = new Buffer(array_buffer); does not copy your typed array, it creates a Buffer representation that uses the same ArrayBuffer as the storage:

> var arr = new Uint8Array(6); var buf = new Buffer(arr.buffer); buf
<Buffer 00 00 00 00 00 00>
> arr[0] = 2; buf
<Buffer 02 00 00 00 00 00>

Edit: example fixed, there was a mistake in the initial version.

var another_node_buffer = new Buffer(node_buffer); copies your buffer to a new one and uses its own storage, which is not guaranteed to be of the same length as your data for performance reasons.

See https://nodejs.org/api/buffer.html#buffer_new_buffer_buffer and https://nodejs.org/api/buffer.html#buffer_class_slowbuffer.

[ChALkeR]

Btw, is new Buffer(array_buffer) documented?

[jasnell]

@ChALkeR ... not actually seeing that behavior at all:

bash-3.2$ ./node -v
v6.0.0-pre
bash-3.2$ ./node 
> const arr = new Uint8Array(6);
undefined
> arr
Uint8Array [ 0, 0, 0, 0, 0, 0 ]
> const buf = new Buffer(arr);
undefined
> buf
<Buffer 00 00 00 00 00 00>
> arr[0] = 2
2
> buf
<Buffer 00 00 00 00 00 00>
> 

See: https://nodejs.org/dist/latest-v5.x/docs/api/buffer.html#buffer_buffer

Edit... hmm ok, nevermind... went back and re-read the example above... it's a bit inconsistent..

const arr = new Uint8Array(6); 
const buf = new Buffer(arr.buffer);
arr[0] = 2;
console.log(buf);
  // Prints: <Buffer 02 00 00 00 00 00>

[ChALkeR]

@jasnell Sorry, there was a copy-paste error in my example (when I tried to fit it into two repl lines). Fixed.

[Fishrock123]

cc @trevnorris

[trevnorris]

So the confusion seems to be that new Buffer(node_buffer) should essentially perform a node_buffer.slice() operation instead of copying the data into a new Buffer instance.

How this works isn't likely to change. Do we need better explanation in our documentation?

[ChALkeR]

@trevnorris That and the missing documentation for new Buffer(array_buffer).

[jasnell]

I believe this is addressed with the recent commits. Closing

[jbergknoff]

I came across this issue myself after investigating a memory corruption issue in https://github.com/mozilla/pdf.js/.

Here's behavior that I consider very surprising:

> var x = new Buffer([1]);
undefined
> new Buffer(x).length
1
> new Buffer(x.buffer).length
8192

@jasnell can you elaborate about which commits address this and/or how it's been addressed. Thank you.

[trevnorris]

@jbergknoff That is expected. Buffer.poolSize is the number of bytes to allocate to a shared pool. Any allocation that's <= Buffer.poolSize / 2 in size will be allocated from this pool. In your example do console.log(x.byteOffset) and you'll see how many bytes into the pool that allocation starts. node has been doing this since v0.3.

[jbergknoff]

@trevnorris thank you for the clarification. With respect, it's only expected if one is intimately familiar with the implementation details of Node's Buffer. I've used Buffer somewhat extensively, but I was unaware it even exposed a .buffer property until I had to dig into the typed array usage in the library I mentioned above. Perhaps the documentation could be expanded on this point.

[ChALkeR]

@jbergknoff https://nodejs.org/api/buffer.html#buffer_buffers_and_typedarray