diff --git a/NETWORK_SETUP.md b/NETWORK_SETUP.md index 78dc6e4e2..557d21293 100644 --- a/NETWORK_SETUP.md +++ b/NETWORK_SETUP.md @@ -10,11 +10,7 @@ gives the most flexibility and outlines the way we setup networks. Starting a Filecoin network requires some special setup. To produce blocks on the Filecoin network, and therefore process messages, a miner on the network must exist with power. For the start of the network this power is given to a miner during -the creation of the genesis. The power given to the miner isn't just a number, the miner is still required to -submit Window PoSt proofs as well as Winning PoSts for block production, there is no special casing for these miners. (The sectors -can be special in one way though, because we can freely construct the genesis there is no pre-commit process). -This requires that sectors exist. The process of creating these sectors before the network is created is called pre-sealing, -and is done through the lotus-seed tool. More than one miner can be created during this process. +the creation of the genesis. ## Basic steps @@ -32,6 +28,103 @@ Binaries required: - lotus - lotus-miner + +# Simple Example + +This example documents how to set up a Filecoin Network called `demonet` with a simple network configuration. + +A minimally viable Lotus network is comprised of: +* Lotus "miner" with presealed sectors +* Lotus "bootstrap" node(s) +* Genesis block +* Faucet + +## Getting Ready + +To follow this demo, you should be on a Linux or Mac computer with at least 16GB memory. + +We'll need + +## Compiling Binaries + +At this time, there are several [lotus](https://github.com/filecoin-project/lotus) files that need edited to create a new network. + +For this example we'll start by copying and editing network parameters from a pre-defined network - `butterflynet`. + +``` +cp buid/params_butterfly.go build/params_demo.go +``` + +We'll need to replace the top two lines with our new network name and some others basic parameters by editing `build/params_demo.go` +Network upgrade epochs get set to negative integers which will signal all upgrades to be skipped and the network will start at the latest network version. +``` +// +build demonet +... +const BootstrappersFile = "demonet.pi" +const GenesisFile = "demonet.car" + +const UpgradeBreezeHeight = -1 +const BreezeGasTampingDuration = -2 +const UpgradeSmokeHeight = -2 +const UpgradeIgnitionHeight = -3 +const UpgradeRefuelHeight = -4 + +var UpgradeAssemblyHeight = abi.ChainEpoch(-5) + +const UpgradeTapeHeight = -6 +const UpgradeLiftoffHeight = -7 +const UpgradeKumquatHeight = -8 +const UpgradeCalicoHeight = -9 +const UpgradePersianHeight = -10 +const UpgradeClausHeight = -11 +const UpgradeOrangeHeight = -12 +const UpgradeTrustHeight = -13 +const UpgradeNorwegianHeight = -14 +const UpgradeTurboHeight = -15 +const UpgradeHyperdriveHeight = -16 + +... +BuildType = BuildDemonet +... +const BootstrapPeerThreshold = 1 + +``` + +Configuration and explanation of all parameters is outside the scope of this document. + +We'll also need to add a new constant and `BuildType` case in `build/version.go` +``` +// +build demonet + +const( + ... + BuildDemonet = 0x8 +) +... + case BuildDemonet: + return "+demonet" +``` + +Also we need to make sure mainnet params aren't used for demonet. In `build/params_mainnet` add + +``` +// +build !demonet +``` + +Finally we should add a `make` target in `Makefile` + +``` +demonet: GOFLAGS+=-tags=demonet +demonet: build-devnets +``` + +Once we've done all this, we can finally compile our binaries +``` +make clean deps demonet install +``` + +Optionally move `lotus-seed` and `lotus-shed` to your executable path. + ## Lotus Shed The `lotus-shed` binary is where lotus keeps all of our tool programs. @@ -49,7 +142,7 @@ lotus-shed keyinfo new bls This will create a file called `libp2p-host-
.keyinfo` which will contain a keyinfo json object. ``` -lotus-shed keyinfo new libp2p +lotus-shed keyinfo new libp2p-host ``` ### Importing keys to lotus repository @@ -72,6 +165,10 @@ lotus-shed fetch-params --proving-params ## Lotus Seed +Creating a genesis file to start a new network requires that sectors exist for genesis miners. +The process of creating these sectors before the network is created is called pre-sealing, +and is done through the `lotus-seed` tool. + The `lotus-seed` binary is used to generate pre-sealed sectors for network bootstrap, as well as constructing the network json configuration use to generate the genesis. @@ -83,25 +180,10 @@ meet the minimum `ConsensusMinerMinMiners` value, pre-sealed sectors can be reus For each miner wanted during the initial setup of a new network, sectors need to be pre-sealed. ``` -lotus-seed pre-seal --miner-addr t01000 --key ./pre-seal-t01000.key --sector-size --num-sectors 4 --sector-offset 0 -``` - -The pre-sealing process can be run in parallel by suppling proper `--sector-offset` values. In a difference process the -output of this process can be merged together to create a single metadata file for the miner. - -``` -lotus-seed --sector-dir /storage//pre-seal-0 pre-seal ... --num-sectors 4 --sector-offset 0 -lotus-seed --sector-dir /storage//pre-seal-1 pre-seal ... --num-sectors 4 --sector-offset 4 -lotus-seed --sector-dir /storage//pre-seal-2 pre-seal ... --num-sectors 4 --sector-offset 8 +lotus-seed pre-seal --miner-addr t01000 --sector-size 512MiB --num-sectors 4 --sector-offset 0 --key bls-.keyinfo ``` -Note: The - -Note: Specifying a key is not required during lotus-seed (a key will be generated). When running parallel pre-sealing -using the `--sector-offset` flag keys should be generated beforehand and pass it into the `lotus-seed` process. However, -the sectors are not tied to any particilar key till the genesis is created. The key can be modified at anytime prior by -editing the `pre-seal-.json` files, or better yet, waiting till after running the `aggregate-manifests` command which -will produce a single file making it easier to edit the owner and worker keys. +This will produce a file that looks something like: ``` cat pre-seal-t01000.json @@ -118,35 +200,18 @@ cat pre-seal-t01000.json } ``` -TODO: There should be a command to change owner and work addresses at some point in the process, probably on the genesis network config? - -NOTE: pre-seal does not generate t_aux files which are required to exist along side every p_aux file. You will have to add these later -TODO: THere should be a command to do this or pre-seal should do it automatically. +Note: Specifying a key is not required during lotus-seed (a key will be generated). When running parallel pre-sealing +using the `--sector-offset` flag keys should be generated beforehand and pass it into the `lotus-seed` process. However, +the sectors are not tied to any particilar key till the genesis is created. The key can be modified at anytime prior by +editing the `pre-seal-.json` files. If performing concurrent sector pre-sealing, you can wail until after running the `aggregate-manifests` command which +will produce a single file making it easier to edit the owner and worker keys. -### Merging pre-seal metadata of parallel pre-seals -If run a parallel pre-seal, sector metadata will need to be merged (`pre-seal-.json`) together into a single -file. This file will then be used to construct the network configuration file. +[TODO]: <> (There should be a command to change owner and work addresses at some point in the process, probably on the genesis network config?) -``` -lotus-seed aggregate-manifests \ - /storage//pre-seal-0/pre-seal-.json \ - /storage//pre-seal-1/pre-seal-.json \ - /storage//pre-seal-3/pre-seal-.json > /storage//pre-seal-.json -``` +NOTE: pre-seal does not generate t_aux files which are required to exist along side every p_aux file. You will have to add these later -NOTE: Each of these lines updates some value, the first 3 lines (starting with the `$MinerId`) are probably not needed -``` -/usr/local/bin/lotus-seed aggregate-manifests {{ lotus_miner_presealed_sectors | product([metadatafile]) | map('join', '/') | join(' ') }} \ -| jq --arg Addr "{{ lotus_miner_wallet_address }}" --arg MinerId "{{ lotus_miner_addr }}" --arg VerifiedDeal "{{ lotus_miner_verified_deals | lower }}" ' - .[$MinerId].Owner = $Addr - | .[$MinerId].Worker = $Addr - | .[$MinerId].ID = $MinerId - | .[$MinerId].Sectors[].Deal.Client = $Addr - | .[$MinerId].Sectors[].Deal.VerifiedDeal = ($VerifiedDeal == "true") - | .[$MinerId].Sectors[] |= (.Deal.Label = .CommR."/") -' > "{{ lotus_miner_presealed_metadata }}" -``` +[TODO]: <> (There should be a command to do this or pre-seal should do it automatically.) ### Generate network configuration @@ -154,7 +219,7 @@ Every network requires a genesis file to initialize from. This section will cove which will later be used to generate the `genesis.car` file itself. ``` -lotus-seed genesis new --network-name /storage//genesis.json +lotus-seed genesis new --network-name demonet genesis.json ``` #### Setting a network start time @@ -164,7 +229,7 @@ be defined to reduce the network mining race effect. The `Timestamp` in the genesis configuration is a unix epoch. -TODO: This should be a command in lotus-seed +[TODO]: <> (This should be a command in lotus-seed) ``` # GENESISDELAY is a time in seconds added to the current time to delay the network start by some amount of time @@ -184,58 +249,52 @@ mv ${GENESISTMP} "/storage//genesis.json" For each miner with pre-sealed sectors that should be included in the network will need to have their metadata file added to the genesis. +In our case we'll just include a single miner. ``` -lotus-seed genesis add-miner /storage//genesis.json /storage//pre-seal-.json +lotus-seed genesis add-miner genesis.json ~/.genesis-sectors/pre-seal-t01000.json ``` -#### Adding additional accounts - -Any additional accounts which should have preallocated funds in the network can be added by updating the `Accounts` -field. +### Generating the genesis car file -You will want to keep this amount of funds preallocated through additional accounts to a minimum as all funds in these -accounts will count towards the circulating supply. Having a high ciruclating supply will result in high sector pledge -fees. For this reason, it's better to allocated all of the funds to a single actors, and set that actor to the account ID -of 90, which will not be counted towards the ciruclating supply. For public test networks, this account would be called the -faucet. +``` +lotus-seed genesis car --out demonet.car genesis.json +``` -TODO: This should be a command in lotus-seed to add accounts -TODO: There needs to a way to set the actor ID 90 account. +Optionally copy `demonet.car` to `build/genesis/` and rebuild lotus, lotus-miner and lotus-worker to distribute to others. -``` -# OWNER is a wallet address, use lotus-shed keyinfo --format '{{ .Address }}' .keyinfo -OWNER="" +## Initializing pre-sealed miners -# BALANCE is the amount of AttoFIL (10^-18 FIL) which will be preallocated to the account (wallet) -# 1 FIL == 1000000000000000000 AttoFIL -BALANCE= +Initializing requires running both the lotus daemon and a storage miner. Only a single miner is required to start +a network, but all pre-sealed miners for the network should be initialized together. It's not strictly required though. -GENESISTMP=$(mktemp) +There are no special flags we pass to the lotus daemon for these miners, besides the `--genesis` flag. -jq --arg Owner ${OWNER} --arg Balance ${BALANCE} \ - '.Accounts |= . + [{Type: "account", Balance: $Balance, Meta: {Owner: $Owner}}]' \ - < "/storage//genesis.json" > ${GENESISTMP} -mv ${GENESISTMP} "/storage//genesis.json" -``` +One miner must be marked the genesis miner.The genesis miner is the first miner to run and will mine the initial blocks +required for initialization of all other. Technically the other miners can be initialized after the genesis miner is +actually running the chain the chain is progressing. However, we always initialize them together and allow for a full +network initialization before actually running the miner. -### Generating the genesis.car +For each pre-sealed set of sectors, a `--pre-sealed-sectors` flag must be passed. In this example we are only using one +directory. -To generate the genesis car file, starting a lotus daemon is required. The `genesis.car` file will be created along side -the `genesis.json` file at `/storage//genesis.car` (this is a detail of this script). +Note: All pre-sealed miner daemons must be connected, this can be done by manually connecting them by running a +`lotus net connect` from all miners to the genesis miner's daemon. However, we recommend that you setup proper bootstrap +nodes and compile their addresses into the lotus daemon using the `build/bootstrap/demonet.pi`. -``` -GENESIS_JSON="/storage//genesis.json" -GENPATH=$(mktemp --suffix=genesis -d) +After initialization the only thing requires is to start the storage miners themselves by running `lotus-storage-miner run`. -cp "${GENESIS_JSON}" "${GENPATH}/genesis.json" +### Genesis miner -lotus-seed genesis car \ - --out="${GENPATH}/devnet.car" \ - "${GENPATH}/genesis.json" +Note: There is nothing special about the `t01000` address here. It has just become practice to make it the genesis miner -cp ${GENPATH}/devnet.car $(dirname $GENESIS_JSON)/genesis.car -rm -rf ${GENPATH} +``` +lotus daemon --genesis=demonet.car --bootstrap=false & +lotus-miner init --actor t01000 \ + --sector-size=512MiB \ + --pre-sealed-metadata=~/.genesis-sectors/pre-seal-t01000.json \ + --pre-sealed-sectors=~/.genesis-sectors \ + --nosync --genesis-miner ``` ## Running bootstrap nodes @@ -248,15 +307,15 @@ Bootstrap nodes are just lotus daemons, ideally with a pre-generated libp2p host without having to distribute new multiaddrs. Bootstrap nodes must all be publicly dial-able and have a static port, as lotus by default uses a random port assigned by the kernel. -To compile bootstrap multiaddrs into the `lotus` binary, place the multiaddrs in `./build/bootstrap/bootstrappers.pi`. +### Start bootstrap daemon -Bootstrap nodes should also run the bootstrap profile by specifying it on the daemon commands. +Start the bootstrap daemon with a profile. ``` lotus daemon --profile=bootstrapper ``` -Optically you may want to increase the `ConnMgrLow` and `ConnMgrHigh` values to increase. +Optimally you may want to increase the `ConnMgrLow` and `ConnMgrHigh` values. The `config.toml` should be located under `$LOTUS_PATH/config.toml`. @@ -276,67 +335,26 @@ description=Lotus daemon firewall rules ports=1347/tcp ``` -### Tipset Thresholds +### Update bootstrap list -Before lotus will start syncing a tipset, lotus requires that is has seen the tipset from a certain number of peers. -This value is defined by the build constant `BootstrapPeerThreshold`. When this value is set to any number greather -than one (1), starting a network runs into issues. This is because at the beggining of the network, there is only a -single peer that actually is producing blocks and sharing the tipset. Therefore no peers will try to sync the tipset. +Get the bootstrap node's id -To resolve this issue, the environment var `LOTUS_SYNC_BOOTSTRAP_PEERS` needs to be set to `1` on at least a number of peers -equal to the value of `BootstrapPeerThreshold`. - -See https://github.com/filecoin-project/lotus/issues/5474 for more information. - -## Initializing pre-sealed miners - -Initializing requires running both the lotus daemon and a storage miner. Only a single miner is required to start -a network, but all pre-sealed miners for the network should be initialized together. It's not strictly required though. - -There are no special flags we pass to the lotus daemon for these miners, besides the `--genesis` flag. - -One miner must be marked the genesis miner.The genesis miner is the first miner to run and will mine the initial blocks -required for initialization of all other. Technically the other miners can be initialized after the genesis miner is -actually running the chain the chain is progressing. However, we always initialize them together and allow for a full -network initialization before actually running the miner. - -For each pre-sealed set of sectors, a `--pre-sealed-sectors` flag must be passed. In this example we are only using three -directories. - -Note: All pre-sealed miner daemons must be connected, this can be done by manually connecting them by running a -`lotus net connect` from all miners to the genesis miner's daemon. However, we recommend that you setup proper bootstrap -nodes and compile their addresses into the lotus daemon using the `build/bootstrap/bootstrappers.pi`. - -After initialization the only thing requires is to start the storage miners themselves by running `lotus-storage-miner run`. - -### Genesis miner - -Note: There is nothing special about the `t01000` address here. It has just become practice to make it the genesis miner +``` +lotus net id +``` +Let's create a bootstrap node list at `build/bootstrap/demonet.pi` +with a single `localhost` multiaddr using the libp2p key we created ``` -lotus-storage-miner init --actor t01000 \ - --sector-size \ - --pre-sealed-metadata /storage/t01000/pre-seal-t01000.json \ - --pre-sealed-sectors /storage/t01000/pre-seal-0 \ - --pre-sealed-sectors /storage/t01000/pre-seal-1 \ - --pre-sealed-sectors /storage/t01000/pre-seal-2 \ - --nosync --genesis-miner +/dns4/localhost/tcp/1347/p2p/ ``` -### Other miners +Once you've done this, you need to rebuild lotus again and distribute as needed. -Other miners have a similar initialization process, they just should not specify the `--genesis-miner` flag. This will -stop them from forking the network immediately by trying to mine their own chain. +As mentioned, using pre-generated keys is recommended outside of this demo. -``` -lotus-storage-miner init --actor \ - --sector-size \ - --pre-sealed-metadata /storage//pre-seal-.json \ - --pre-sealed-sectors /storage//pre-seal-0 \ - --pre-sealed-sectors /storage//pre-seal-1 \ - --pre-sealed-sectors /storage//pre-seal-2 \ - --nosync -``` + +# Advanced Usage ## Other network configuration @@ -346,9 +364,8 @@ of the specs-actors code itself. Most notability are the supported sector sizes, minimum miner power, and block time. Currently these should be changed by editing `build/params_testnet.go` of the lotus source code. ------------------------------- -### Verified Registery +## Verified Registery The Verified Registery can be interactived through the lotus-shed tool using signed messages. The address is set during genesis creation. To be able to use the verified registery on a network, generate a key and replace the `Signers` array with @@ -374,7 +391,90 @@ the address. } ``` -### Limits on number of sectors +## Adding additional accounts + +Any additional accounts which should have preallocated funds in the network can be added by updating the `Accounts` +field. + +You will want to keep this amount of funds preallocated through additional accounts to a minimum as all funds in these +accounts will count towards the circulating supply. Having a high ciruclating supply will result in high sector pledge +fees. For this reason, it's better to allocated all of the funds to a single actors, and set that actor to the account ID +of 90, which will not be counted towards the ciruclating supply. For public test networks, this account would be called the +faucet. + +[TODO]: <> (This should be a command in lotus-seed to add accounts) +[TODO]: <> (There needs to a way to set the actor ID 90 account.) + +``` +# OWNER is a wallet address, use lotus-shed keyinfo --format '{{ .Address }}' .keyinfo +OWNER="" + +# BALANCE is the amount of AttoFIL (10^-18 FIL) which will be preallocated to the account (wallet) +# 1 FIL == 1000000000000000000 AttoFIL +BALANCE= + +GENESISTMP=$(mktemp) + +jq --arg Owner ${OWNER} --arg Balance ${BALANCE} \ + '.Accounts |= . + [{Type: "account", Balance: $Balance, Meta: {Owner: $Owner}}]' \ + < "/storage//genesis.json" > ${GENESISTMP} +mv ${GENESISTMP} "/storage//genesis.json" +``` + + +## Other miners + +Other miners have a similar initialization process, they just should not specify the `--genesis-miner` flag. This will +stop them from forking the network immediately by trying to mine their own chain. + +``` +lotus-storage-miner init --actor \ + --sector-size \ + --pre-sealed-metadata /storage//pre-seal-.json \ + --pre-sealed-sectors /storage//pre-seal-0 \ + --pre-sealed-sectors /storage//pre-seal-1 \ + --pre-sealed-sectors /storage//pre-seal-2 \ + --nosync +``` + +## Pre-sealing Sectors in Parallel + +The pre-sealing process can be run in parallel by suppling proper `--sector-offset` values. In a difference process the +output of this process can be merged together to create a single metadata file for the miner. + +``` +lotus-seed --sector-dir /storage//pre-seal-0 pre-seal ... --num-sectors 4 --sector-offset 0 +lotus-seed --sector-dir /storage//pre-seal-1 pre-seal ... --num-sectors 4 --sector-offset 4 +lotus-seed --sector-dir /storage//pre-seal-2 pre-seal ... --num-sectors 4 --sector-offset 8 +``` + +### Merging pre-seal metadata of parallel pre-seals + +If run a parallel pre-seal, sector metadata will need to be merged (`pre-seal-.json`) together into a single +file. This file will then be used to construct the network configuration file. + +``` +lotus-seed aggregate-manifests \ + /storage//pre-seal-0/pre-seal-.json \ + /storage//pre-seal-1/pre-seal-.json \ + /storage//pre-seal-3/pre-seal-.json > /storage//pre-seal-.json +``` + +NOTE: Each of these lines updates some value, the first 3 lines (starting with the `$MinerId`) are probably not needed +``` +/usr/local/bin/lotus-seed aggregate-manifests {{ lotus_miner_presealed_sectors | product([metadatafile]) | map('join', '/') | join(' ') }} \ +| jq --arg Addr "{{ lotus_miner_wallet_address }}" --arg MinerId "{{ lotus_miner_addr }}" --arg VerifiedDeal "{{ lotus_miner_verified_deals | lower }}" ' + .[$MinerId].Owner = $Addr + | .[$MinerId].Worker = $Addr + | .[$MinerId].ID = $MinerId + | .[$MinerId].Sectors[].Deal.Client = $Addr + | .[$MinerId].Sectors[].Deal.VerifiedDeal = ($VerifiedDeal == "true") + | .[$MinerId].Sectors[] |= (.Deal.Label = .CommR."/") +' > "{{ lotus_miner_presealed_metadata }}" +``` + + +## Limits on number of sectors There is an upper bound to the number of sectors a miner can have. This doesn't really affect 32/64 sector miners, but the smaller test sectors have a limit of 2 sectors per partition and because lotus submits all parittions as a single @@ -441,3 +541,15 @@ for MINER_ADDR in ${MINERS[@]}; do done done ``` + +## Tipset Thresholds + +Before lotus will start syncing a tipset, lotus requires that is has seen the tipset from a certain number of peers. +This value is defined by the build constant `BootstrapPeerThreshold`. When this value is set to any number greather +than one (1), starting a network runs into issues. This is because at the beggining of the network, there is only a +single peer that actually is producing blocks and sharing the tipset. Therefore no peers will try to sync the tipset. + +To resolve this issue, the environment var `LOTUS_SYNC_BOOTSTRAP_PEERS` needs to be set to `1` on at least a number of peers +equal to the value of `BootstrapPeerThreshold`. + +See https://github.com/filecoin-project/lotus/issues/5474 for more information.