Migrate libraries into core monorepo

[Gudahtt]

We would like to migrate most MetaMask libraries into this monorepo to make them easier to maintain. We may want to exclude libraries that are:

• Experimental
• Deprecated, or not encouraged to be used in our codebase anymore
• Not used directly in a MetaMask client
• Intended primarily for external developers integrating with MetaMask
• Specific to extensions or React Native
• Too noisy/high-volume
• Deployed both as a website and an npm package
• Controllers (these are represented by Merge extension and mobile controllers #700)
• Conceptually distinct in some way (this one is subjective)

Here is the list of libraries that seem like a good fit today:

• https://github.com/MetaMask/abi-utils
• https://github.com/MetaMask/eth-block-tracker
• https://github.com/MetaMask/eth-hd-keyring
• https://github.com/MetaMask/eth-json-rpc-filters
• https://github.com/MetaMask/eth-json-rpc-infura
• https://github.com/MetaMask/eth-json-rpc-middleware
• https://github.com/MetaMask/eth-json-rpc-provider
• https://github.com/MetaMask/eth-ledger-bridge-keyring
• https://github.com/MetaMask/eth-method-registry
• https://github.com/MetaMask/eth-phishing-detect (but just the detector, not the config)
• https://github.com/MetaMask/eth-rpc-errors
• https://github.com/MetaMask/eth-sig-util
• https://github.com/MetaMask/eth-simple-keyring
• https://github.com/MetaMask/eth-token-tracker
• https://github.com/MetaMask/eth-trezor-keyring
• https://github.com/MetaMask/etherscan-link
• https://github.com/MetaMask/json-rpc-engine
• https://github.com/MetaMask/json-rpc-middleware-stream
• https://github.com/MetaMask/key-tree ? (check in with Snaps team first)
• https://github.com/MetaMask/metamask-eth-abis
• https://github.com/MetaMask/nonce-tracker
• https://github.com/MetaMask/providers
• https://github.com/MetaMask/rpc-errors
• https://github.com/MetaMask/scure-bip39 This is a fork, so we might not want to do this
• https://github.com/MetaMask/utils

See #1079 (comment) for instructions on accomplishing this.

[mcmire]

I propose we focus on the following libraries for Q3:

• @metamask/utils
• @metamask/json-rpc-engine
• @metamask/eth-json-rpc-provider (not in the list above because it was created after the above list was built)
• @metamask/json-rpc-middleware-stream

[mcmire]

I went through the list of existing repos and found some that were omitted from the list above:

• browser-passworder — Is this because it's extension only?
• Snap-related libraries (eth-snap-keyring, keyring-snaps-registry, snap-simple-keyring, snaps-registry) — is this because Snaps owns these?
• object-multiplex (used by providers) — Is it because it's not used directly by the wallet?
• post-message-stream — Is this because it's extension only?
• ppom-validator — Is this because it's extension only?
• rpc-errors, safe-event-emitter, eth-json-rpc-provider, key-tree, swappable-obj-proxy — Are these missing because they were created after the list above was compiled?
• web3-stream-provider — Is this because it's extension only?

[mcmire]

UPDATE October 17, 2023:

Before starting work on future library migrations, please make sure to consult @MajorLift's migration checklist here. The instructions here only cover step B.0 in this checklist.

---

One thing we want to absolutely make sure to do when we migrate a library to this repo is to preserve all of the Git history.

I experimented with this using @metamask/utils and came up with a process. At a high level, we need to:

1. Clone the repo for the library somewhere.
2. Rewrite the entire history of the library as though the entire project had been located in a merged-packages/<library-name> directory all along. (You will soon see why.)
3. Navigate to core.
4. Add the library repo as a (local) remote, and fetch its history.
5. Create a merge commit that connects the primary branch of the library with core's main branch (ignoring unrelated history).
6. Done! Now we can submit a pull request.

Now we see why we performed step 2: because we want the library to initially be located in core under merged-packages/<library-name>. This directory is special in that ESLint and Prettier will ignore it. This allows us to follow up with another pull request that moves the library into packages and incorporate it into the release workflow.

Here is a list of detailed steps:

1. Install git-filter-repo. This tool is like Git's filter-branch command, but much easier to use and much less error-prone.
2. Navigate to a temporary directory: cd /tmp
3. Clone the repo for the library you want to migrate: git clone https://github.com/MetaMask/<repo-name> (e.g. <repo-name> = utils). Do NOT use an existing clone.
4. cd into the newly cloned repo.
5. Run git filter-repo --to-subdirectory-filter merged-packages/<package-name> (e.g. <package-name> = utils). This will rewrite all history to move all files to merged-packages/<package-name>. (This is why we're making a fresh clone in a temporary directory — this action is irreversible.)
6. cd to wherever you keep the core repo on your machine.
7. Add the library as a remote: git remote add <package-name> /tmp/<package-name>.
8. Fetch history: git fetch <package-name> --no-tags
9. Make a new branch: git checkout -b migrate-<package-name>
10. Merge the library into the monorepo: git merge --allow-unrelated-histories <package-name>/<primary-branch> (e.g. <primary-branch> = main)
11. Make a pull request.

The resulting pull request should look like this: #1520

[desi]

browser-passworder, post-message-stream, web3-stream-provider — Not included because it's extension only
Snap-related libraries (eth-snap-keyring, keyring-snaps-registry, snap-simple-keyring, snaps-registry) —not included because Snaps only.
object-multiplex, swappable-obj-proxy, and safe-event-emitter - Not included because its not really wallet related, they are more general purpose
ppom-validator — Team decided to keep out of core for now - maybe revisit