Usage in Browser with alternative Buffer

[srsudar]

I would love to be able to use this in a browser using Feross's buffer module. It has API parity with node's Buffer. I thought I could get around this with smartBuffer.from(new Buffer(1024)), but I am getting an error: Invalid buffer provided in SmartBufferOptions.

Is there some way to specify an alternative Buffer that I am missing?

[JoshGlazebrook]

So you should be using:

SmartBuffer.fromBuffer(theBuffer);

What does your setup look like for how you're using the module in the browser?

Setting up a simple webpack config with babel-loader and the es2015 preset, it seems to work just fine.

import { Buffer } from 'buffer';
import { SmartBuffer } from 'smart-buffer';


const someArray = [1,2,3,4,5,6,7,8,9,0];

const buff = Buffer.from(someArray);
console.log(buff);

const sBuff = SmartBuffer.fromBuffer(buff);
console.log(sBuff);

[srsudar]

Does your example work if you import from 'buffer/'? The trailing slash makes a difference according to Feross, at least with using require(). I am using browserify. If I do require('buffer').Buffer, my tests pass but it fails in the browser. If I add the trailing slash, the tests fail.

The following fails for me:

const SmartBuffer = require('smart-buffer').SmartBuffer;
const Buffer = require('buffer/').Buffer;

let buff = Buffer.from('hello');
let smartBuff = SmartBuffer.fromBuffer(buff);

I am using browserify to bundle everything and then am running as a Chrome App. My tests fail using tape, and a Polymer Chrome App fails. If you want to see a full example, take a look at this branch. cd into the chromeapp/ directory and run grunt tape.

[JoshGlazebrook]

It's definately loading Feross' buffer with my webpack version.

From what I can tell from the test failures in your code, the Buffer that SmartBuffer sees is not the same as the Buffer that is being passed in. It's failing an instanceof test, so for some reason whatever global Buffer SmartBuffer is seeing is different.

Not sure if it's related to this: #13 (comment)

[JoshGlazebrook]

Also found this: feross/buffer#40

I might have to use https://github.com/feross/is-buffer for checking if something is a Buffer. Unless you can figure something else out? I'm not sure why this is an issue.

[srsudar]

Man am I ever puzzled. It looks like for whatever reason the Buffer that is being resolved by SmartBuffer is not the same as the Buffer I am passing in.

instanceof checks on these two fail.

It looks like the version of Buffer SmartBuffer sees is from Buffer 4.x. It isn't clear to me why this is, but I think it is either:

1. browserify says it tries to be clever when resolving Buffer. Maybe it is giving SmartBuffer the old version.
2. One of my dependencies in package.json has buffer-4.9.1 as a transitive dependency. Maybe that is for some reason being handed to SmartBuffer.

Using Buffer.isBuffer() instead of an instanceof makes sense to me, as it is also available on Node's Buffer. instanceof strikes me as unreliable, but the articles I've read giving it a bad reputation might be outdated by now.

I'm not sure what to expect of SmartBuffer. This seems more like an artifact of my build that Buffer.isBuffer() would fix, but the problem could still be there. The ensure capacity checks you do use things like Buffer.copy(). If my build is giving you an indeterminate Buffer object to work with, problems could still be lurking if I pass in a 5.x Buffer and get back a 4.x Buffer. I'm not familiar enough with Node to know if things like isBuffer() exist to get around this sort of thing, if other modules let you pass in a class object to use, or if there is some other solution.

[JoshGlazebrook]

So internally Buffer.isBuffer() seems to just use instanceof, so that won't really help. SmartBuffer should just be using whatever the global Buffer is defined as.

Since it's working with the webpack setup I put together, I think it definitely is coming down to what Browserify is choosing to do, but I've never really used it.

[JoshGlazebrook]

So I ran this through browserify:

var Buffer = require('buffer').Buffer;
var SmartBuffer = require('smart-buffer').SmartBuffer;

var someArray = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0];

var buff = Buffer.from(someArray);
console.log(buff);

var sBuff = SmartBuffer.fromBuffer(buff);
console.log(sBuff);

And it seems to have worked since the same Buffer (feross/buffer) is being passed to SmartBuffer.

https://gist.github.com/JoshGlazebrook/594711741c285131a24e92d441ba0cbb

But! If I instead require('buffer/').Buffer above ^, it passes SmartBuffer a different Buffer than the one being used in that file.

Here is a diff between the two:

https://www.diffchecker.com/nib24sRT

[srsudar]

Interesting.... I believe what is happening is that require('buffer/') gets the manually installed version. Without the trailing slash, or referring only to the Buffer global with no require, browserify interprets it as a call to node's default buffer module, which it then wraps with a different version.

Browserify exposes an option, insertGlobalVars, to control the version of the libraries it swaps out. I could use this to fix the problem when running in a browserified bundle. Unfortunately this still wouldn't fix all my problems, as to test my code I'm using tape, which runs in a node environment and isn't browserified. This would then leave my own code operating against feross/buffer, while SmartBuffer would be running against Node's own Buffer. This causes my tests to fail, but the code to work after it is browserified.

However, looking again at the feross/buffer README, it sounds like I have been using it incorrectly. Several times in the README it says that, if using browserify, either rely on the Buffer global or just require('buffer').

Literally the first line:

With browserify, simply require('buffer') or use the Buffer global and you will get this module.

Later:

As mentioned above, require('buffer') or use the Buffer global with browserify and this module will automatically be included in your bundle.

It sounds like the thing to do is rely on node's Buffer while testing in a node environment, and then let browserify handle preparing it for the browser. I've been unwittingly mixing methods. I had an issue a couple months ago where I got into trouble due to mixing Buffer versions. I forget what it was, but I thought the fix was to be ultra vigilant about require('buffer/'). I wonder if it really just stemmed from the same problem...

Yesterday I swapped out SmartBuffer with my own lightweight implementation for a tiny subset of features and got working what I needed. I just put SmartBuffer back in and removed all require('buffer/') calls. Lo and behold, no problems.

Sorry for the confusion, and thanks for the help.

[JoshGlazebrook]

Ah glad you got it working.

Also as a side note, I've been working on v4 for a while now (on and off), which includes a few major breaking changes. Most notably, all of the writeXXX(value, offset) functions now mirror Buffer and overwrite data at the given offset as opposed to inserting. But now I've added .insertXXX(value, offset) functions to behave like how the write functions currently do.

You still can just use .writeXXX(value) without an offset and it will still work without specifying an offset (which is why I made this library in the first place), but it just made more sense to keep inserts separate so you can still overwrite data if needed.

[Jerrylum]

To anyone in the future who got the Webpack 5 Buffer error:

Install https://www.npmjs.com/package/buffer

Then insert the following code into index.tsx. You can insert it to any file, but index is preferred:

import { Buffer } from "buffer";
globalThis.Buffer = Buffer;

Just to let you know, I am using CRACO. It is a better version of CRA. You can install it from here: https://craco.js.org/

I tried something else but it didn't work for me:

Details

I inserted the following code to file `craco.config.js` in the project:

const { addAfterLoader, loaderByName } = require("@craco/craco");

module.exports = async env => {
  const remarkGfm = (await import("remark-gfm")).default;

  return {
    plugins: [
      { plugin: require("./craco-fallback-buffer-plugin.js") },
    ]
  };
};

And created a file craco-fallback-buffer-plugin.js with the following content:

// See: https://github.com/WalletConnect/walletconnect-monorepo/issues/748

module.exports = {
  overrideWebpackConfig: ({ webpackConfig, cracoConfig, pluginOptions, context: { env, paths } }) => {
    /** @type {import('webpack/types'.Configuration)} */
    const newConfig = { ...webpackConfig };

    newConfig.resolve.fallback = webpackConfig.resolve.fallback || {};
    newConfig.resolve.fallback.buffer = require.resolve("buffer/");

    return newConfig;
  }
};

Hope this can help you :)