Haste error messages (React-Native)

[tim-br]

When running this code in react-native

import ethers from 'ethers'

I get this error message:

Module buffer does not exist in the Haste module map.

To fix it I had to revert to 3.0.27

EDIT:
full error message:

error: bundling failed: Error: Unable to resolve module buffer from ./node_modules/ethers/dist/ethers.js: Module buffer does not exist in the Haste module map

This might be related to facebook/react-native#4968
To resolve try the following:

1. Clear watchman watches: watchman watch-del-all.
2. Delete the node_modules folder: rm -rf node_modules && npm install.
3. Reset Metro Bundler cache: rm -rf /tmp/metro-bundler-cache-* or npm start -- --reset-cache. 4. Remove haste cache: rm -rf /tmp/haste-map-react-native-packager-*.
   at ModuleResolver.resolveDependency (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/node-haste/DependencyGraph/ModuleResolution.js:167:1306)
   at ResolutionRequest.resolveDependency (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/node-haste/DependencyGraph/ResolutionRequest.js:80:16)
   at DependencyGraph.resolveDependency (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/node-haste/DependencyGraph.js:237:485)
   at Object.resolve (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/lib/transformHelpers.js:116:25)
   at dependencies.map.result (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/DeltaBundler/traverseDependencies.js:298:29)
   at Array.map ()
   at resolveDependencies (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/DeltaBundler/traverseDependencies.js:294:16)
   at /Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/DeltaBundler/traverseDependencies.js:159:33
   at Generator.next ()
   at step (/Users/tim/Desktop/mobileWorkspace/albums/node_modules/metro/src/DeltaBundler/traverseDependencies.js:239:307)

[ricmoo]

What is Haste?

[skyline75489]

@ricmoo Haste is an RN thing which basically means the module is not found.

[ricmoo]

There should be no buffer modules included at all though. Are you using TypeScript?

For TypeScript, the library should be imported with:

import { ethers } from "ethers";

But I don't see how that could be the problem.

I've never used RN, but can you provide a very simple project which replicates the issue?

[tim-br]

https://github.com/strangepowers/test_ethers_react_native

this is the commit that doesn't work: https://github.com/strangepowers/test_ethers_react_native/commit/94fc5527ae5da099b1c9417133e16775424221e6

I suspect it may have something to do with these:

facebook/react-native#14796
feross/safe-buffer#18

so I tried experimenting with https://www.npmjs.com/package/rn-nodeify using this project as an example: https://github.com/mvayngrib/react-native-crypto , but no luck so far.

[philipp-sorin]

You can inject it via extraNodeModules in rn-cli.config.js
something like

// in rn-cli.config.js 
module.exports = {
//... your configuration 
    extraNodeModules: require('./extra-node-modules-setup')
}

// in extra-node-modules-setup.js
exports.buffer	= require.resolve('buffer'); // buffer is a package from npm 
// .... other extra modules

At least it worked fine for similar issue with web3.js

[ricmoo]

I've been trying to fix this. It seems like the react-native environment overwrites require, even when it locally scoped. For example:

function foo(require) {
    let req = require;
    (function() {
        assert.ok(req == require, "this seems to fail");
    })();
}
foo(require);

It seems like the runtime does very strange things. I'm trying to find a work around.

[ricmoo]

I have found a solution. It is HACKY. It is VERY hacky!

Overview of the problem, The module system for React-Native hijacks any variable with the name require, unless immediately in the scope it is replaced with its the Haste require. This means that the correct require, that we create and coddle very carefully using closures within Browserify is quickly obliterated.

So, easy work around, assign require to another name immediately in EVERY function. Wait, what?! No! That sounds insane.

Well, yes, that would be. Luckily we already have something that has done that for us.

To use ethers with React-Native (for now; I will come up with a better solution in the future using the package.json module), use the following import:

import { ethers } from 'ethers/dist/ethers.min.js';

The minified version of ethers.js has already replaced all instances of the function named "require" with other, less-human-readable (more space-saving) variables names like r and q. Most importantly, it is not the word "require".

Le sigh.

[nikitaeverywhere]

Making ethers.js module-ar will require a lot of refactoring, but it would be much appreciated thing. Even though it is 300kb without GZipping (which is okay), I believe modularity will drastically reduce its size. Many pieces of ethers.js are reusable but unfortunately not modular enough at some places.

Thanks for your work on the project!

P.S.
import { ethers } from 'ethers' (which works) vs import ethers from 'ethers' (which doesn't) is very strange.

[ricmoo]

We actually just finished un-modularizing it. Making things modular makes the thing as a whole larger, in general. What modularization does it make it smaller for people who only require a subset of features, but if you need it all anyways, it is larger. Modularization also makes testing and updating dependencies a nightmare to do with explicit versions, which are important for security.

We solve this by keeping pathes stable within a release.

For example, if you only need keccak256, const { keccak256 } = require("ethers/utils/keccak256"); will make sure you only increase your app size by the size of keccak256 and required dependencies. A lot of work has gone into making the dependency graph tight, with lightweight leaf dependencies. Also why there is an abstract-provider which is basically useless and an abstract-signer; so that the dependency graph for things that need one or the other don't balloon to the full size of the library for simple tasks. Basically once you end up including anything that requires secp256k1 elliptic curve operations, you are better off pulling in the full package. Tree shaking will fix this once we have a module ES6 version in the package.json, but that requires run-time feature shimming, while right now we do fist-time feature shimming. Soon we will focus on it. :)

Within a major version, the paths of functionality will not change.

The reason import { ethers } ... works and import ethers... does not is because we maintain compatibility with ES3 and older versions of node. The ES6 import and TypeScript model for defaults does not play well with others. To support defaults, it would require node 6 users to type var ethers = require('ethers').default, which is cumbersome... Most ES6 and TS folk are already used to typing in the explicit packages they want to import, so this seemed like the least-worst option. It does mean there is a weird ethers.js file that is included and exported under two roots in index.html, but one day, when the median JavaScript environment is modern (by todays standards), a lot of these foibles go away. :)

[nikitaeverywhere]

Thanks for the update, @ricmoo!

You're doing good with possibly the most lightweight, full and stable-functioning Ethereum client, keep it going!

However it took me almost a half of the day to refactor from v3 to v4 of the library (yet not completely), because many things were changed (also, the migration guide is not complete); v3 had a bug with wrong gas price estimation (it couldn't estimate gas for contracts where msg.sender is required by a smart contract function), which was a blocker for us.

[ricmoo]

Thanks!

Pleas let me know any missing parts of the migration guide too. I need to add a mapping of file paths from v3 to v4 too.

The gas limit thing is driving me insane. In v3, I hard-coded 1.5 million gas, which had its own problems. In v4, I use estimate gas, but it seems like if the contract calls an external contract, estimate gas throws its hands in the air and just returns a number that is basically infinite (larger than the block gas limit). I need to figure out some reasonable threashold, that over that, it sets it to that max, or something... But the. It interferes where the gas limit accurately detects the call would fail. :s

The from should be included in estimate gas, but I’ll look into that and verify that too.

[ricmoo]

Oh! Interesting. I see the bug you mean! I’ll fix that ASAP. :)

[ricmoo]

Oh! (again) that’s version 3. I will double check all the code paths in v4 to make sure from is properly propagated.

Basically, I’ll look into this shortly. :)

[zysam]

I have found a solution. It is HACKY. It is VERY hacky!

Overview of the problem, The module system for React-Native hijacks any variable with the name require, unless immediately in the scope it is replaced with its the Haste require. This means that the correct require, that we create and coddle very carefully using closures within Browserify is quickly obliterated.

So, easy work around, assign require to another name immediately in EVERY function. Wait, what?! No! That sounds insane.

Well, yes, that would be. Luckily we already have something that has done that for us.

To use ethers with React-Native (for now; I will come up with a better solution in the future using the package.json module), use the following import:

import { ethers } from 'ethers/dist/ethers.min.js';

The minified version of ethers.js has already replaced all instances of the function named "require" with other, less-human-readable (more space-saving) variables names like r and q. Most importantly, it is not the word "require".

Le sigh.

And how to import the wordlists.

I try to:

require('ethers/dist/wordlist-zh')

Get the error: Cannot read property 'utils' of undefined.

P.S. I run it on react-native.

[ricmoo]

Heya! So, I think I've solved the RN issues... Can you upgrade to 4.0.9 and try:

// Import the required shims
import 'ethers/dist/shims.js';

// Import the ethers library
import { ethers } from 'ethers';

Notes: https://docs.ethers.io/ethers.js/html/cookbook-react.html

[ricmoo]

@zysam I don't think that last push will fix your issue though, I will look into that tomorrow.

[ricmoo]

@zysam To fix your issue, you need to use the React-Native import syntax (the require method is for browsers, which exposes a global object).

import { ethers } from 'ethers';
import { zh_cn } from 'ethers/wordlists';

let mnemonic = ethers.Wallet.createRandom({ locale: zh_cn }).mnemonic;

Let me know if you still have issues with that. :)

[ricmoo]

This should be fixed now entirely. Please try out adding the shims from 4.0.10.

Thanks! :)

[zysam]

@ricmoo

Thank you. It works!

I try to do this:

import 'ethers/dist/shims'

// import 'ethers/wordlists/lang-zh'
import {ethers} from 'ethers'
import * as wordlists from 'ethers/wordlists'

getRandomMnemonic(n = 16, lang = 'en') {
    // Chose the length of your mnemonic:
    //   - 16 bytes => 12 words (* this example)
    //   - 20 bytes => 15 words
    //   - 24 bytes => 18 words
    //   - 28 bytes => 21 words
    //   - 32 bytes => 24 words
    const bytes = this.ethers.utils.randomBytes(n)

    // Select the language:
    //   - en, es, fr, ja, ko, it, zh_ch, zh_tw
    // const wordlists = this.ethers.utils.Wordlist
    const language = wordlists[lang]
    console.log('language:', lang, language)

    const randomMnemonic = this.ethers.utils.HDNode.entropyToMnemonic(bytes, language)
    return randomMnemonic
  }

[ricmoo]

Awesome!! Let me know if there are any more problems. :)

Thanks!

[MicahZoltu]

Is there a reason you don't instead wrap XmlHttpRequest in something like this?

if (window && window.XmlHttpRequest) export = window.XmlHttpRequest
else export = require('XmlHttpRequest')

Note, the above code snippet is meant to get an idea across, it probably isn't valid or correct. The current solution is non-obvious, I only figured it out after a bunch of troubleshooting. The proposed code snippet should (I think) "just work" in either Node or Browser without the user doing anything.

[ricmoo]

In which file?

Any file processed by Browserify needs the file replaced using the shims instead, otherwise the dist file would contain the entire implementation of XmlHttpRequest, despite not requiring it...

I would love to find a better solution though. My plan is to make the module in the package.json point to an ES6 module which pulls in all these dependencies like your suggestion (more or less), since any consumer would have tree-shaking, which could knock these unused dependencies out. :)

[MicahZoltu]

Looks like this solution doesn't work with webpack, which is hard-bundled into Angular. 😢