Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network upgrade support #3781

Merged
merged 110 commits into from
Sep 23, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
d359483
[WIP] Network upgrade support
Stebalien Sep 12, 2020
38f8798
Fix some build errors
magik6k Sep 14, 2020
6809713
Fix vm build
magik6k Sep 14, 2020
683a581
More terraforming in chain/
magik6k Sep 14, 2020
06ec571
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
magik6k Sep 14, 2020
fb3f710
Some post-master merge fixes
magik6k Sep 14, 2020
2088335
rename to simplify merge
Stebalien Sep 14, 2020
02dcb5e
Merge branch 'master' into refactor/net-upgrade
Stebalien Sep 14, 2020
cc4d530
Progress
Stebalien Sep 14, 2020
7fa6c1e
add info method to miner state API
Stebalien Sep 14, 2020
36f920b
progress
Stebalien Sep 15, 2020
4dabab5
state manager progress
magik6k Sep 15, 2020
4419850
state api impl fixes
magik6k Sep 15, 2020
9dad38c
gofmt
magik6k Sep 15, 2020
00c6397
more progress
Stebalien Sep 15, 2020
6bf7976
fix decode
Stebalien Sep 15, 2020
4cd92d8
remove final WithStateTree
Stebalien Sep 15, 2020
53384e8
migrate wallet access
Stebalien Sep 15, 2020
c64f983
migrate StateMinserSectorCount
Stebalien Sep 15, 2020
91e9573
Compile fixes
Stebalien Sep 15, 2020
4e01fad
start multisig abstraction
Stebalien Sep 15, 2020
e1ba4eb
sub reward actor
Stebalien Sep 15, 2020
02bc5fa
add power actor address alias
Stebalien Sep 15, 2020
92471d4
migrate precommit deposit function
Stebalien Sep 15, 2020
454c382
migrate StateMinerInitialPledgeCollateral
Stebalien Sep 16, 2020
05c1153
feat(paych): convert paych actor
hannahhoward Sep 16, 2020
b4ee519
Partial progress towards switching to miner and power interfaces
arajasek Sep 16, 2020
90853e2
Add a boolean HasMinPower to return of GetPower
arajasek Sep 16, 2020
72d19f3
Incremental progress towards using new power state interface
arajasek Sep 16, 2020
ccce1a9
Merge pull request #3877 from filecoin-project/refactor/net-upgrade-p…
magik6k Sep 16, 2020
b530f25
Migrate miner actor
arajasek Sep 16, 2020
7115485
Add a getter for network version
arajasek Sep 17, 2020
80b6994
feat(market): update state diffing for market actor
hannahhoward Sep 17, 2020
81004ea
Merge pull request #3889 from filecoin-project/refactor/net-upgrade-m…
arajasek Sep 17, 2020
b5ba7a0
Miner migration
arajasek Sep 17, 2020
9e48dd2
Fixups
arajasek Sep 17, 2020
b606149
Migrate reward actor
arajasek Sep 17, 2020
e2295c3
Migrate multisig actor
arajasek Sep 17, 2020
691bd9f
feat(markets): complete markets conversion
hannahhoward Sep 17, 2020
7bf165c
Remove miner-related specs actors types from API
arajasek Sep 17, 2020
053cfc1
Migrate verifreg actor
arajasek Sep 17, 2020
31ff523
Get State API almost working
arajasek Sep 17, 2020
e632643
api: Test return types
magik6k Sep 17, 2020
6eda535
Most tests passing
magik6k Sep 17, 2020
68e884e
docsgen
magik6k Sep 17, 2020
82b95e3
cbor gen
magik6k Sep 17, 2020
c40c136
fix: paych To()
dirkmc Sep 17, 2020
18bc59b
uncruft
Stebalien Sep 17, 2020
708aa36
remove dep on v0 miner
Stebalien Sep 17, 2020
ae38970
remove WpostProvignPeriod function
Stebalien Sep 17, 2020
dc58f71
remove unnecessary code
Stebalien Sep 17, 2020
5bcfee0
make market diffs work across version upgrades
Stebalien Sep 17, 2020
b2ee590
improve diff logic
Stebalien Sep 18, 2020
f2a0779
remove specs-actors import
Stebalien Sep 18, 2020
8bd6791
more fixups
Stebalien Sep 18, 2020
daa441b
simplify market diff
Stebalien Sep 18, 2020
37de154
Fixup tests
arajasek Sep 18, 2020
fce423c
Appease the linter and get everything building
arajasek Sep 18, 2020
694463f
More lint fixes
arajasek Sep 18, 2020
70faa36
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
magik6k Sep 18, 2020
ccd0a67
wip fixing wdpost tests
magik6k Sep 18, 2020
c3046f4
Don't use -1 as default sector number
arajasek Sep 18, 2020
fb2b25c
finish upgrading chainwatch
Stebalien Sep 18, 2020
298efa2
fix lint errors
Stebalien Sep 18, 2020
8747c60
abstract over account actor
Stebalien Sep 18, 2020
24df873
rename imports
Stebalien Sep 18, 2020
e60027c
remove todo
Stebalien Sep 18, 2020
8285eda
migrate storage miner info
Stebalien Sep 18, 2020
4a7055c
fix state loading in vm
Stebalien Sep 18, 2020
f741ce6
fixup some more imports
Stebalien Sep 18, 2020
c130806
compile fix
Stebalien Sep 18, 2020
1bf3b49
rename imports to match actors code
Stebalien Sep 18, 2020
35bce5a
revert post changes
Stebalien Sep 18, 2020
7c3f638
Abstract FilterEstimate, PreCommitDepositForPower, and InitialPledgeF…
arajasek Sep 19, 2020
b25dd2a
More correct / fasterer GetSectorsForWinningPoSt
magik6k Sep 20, 2020
ed285f1
Abstract SectorOnChainInfo and SectorPreCommitOnChainInfo
arajasek Sep 20, 2020
a95e34f
Fix build
arajasek Sep 21, 2020
b355eb7
Add compile-time checks that actors interfaces are correctly implemented
arajasek Sep 21, 2020
286fe04
Add some TODOs
arajasek Sep 21, 2020
a307e45
wdpost: Fix TestWDPostDoPost
magik6k Sep 21, 2020
0b5e4a9
Make GetSectorsForWinningPoSt fast again
magik6k Sep 21, 2020
f7934b0
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
magik6k Sep 21, 2020
0256631
non-destructive diff
Stebalien Sep 21, 2020
4cf0c10
optimize sector loading
Stebalien Sep 21, 2020
24ae920
cross-version state tree diff
Stebalien Sep 21, 2020
f9a896f
fix compile
Stebalien Sep 21, 2020
0ab2459
convert multisig to new actor abstractions
Stebalien Sep 21, 2020
a41bf74
fix: remove incorrect variable
Stebalien Sep 21, 2020
916421b
convert lotus-shed balances
Stebalien Sep 21, 2020
4bab784
migrate lotus-shed verifreg to specs-actors abstractions
Stebalien Sep 21, 2020
3f0106c
migrate lotus-shed/genesis-verify to actor abstraction
Stebalien Sep 21, 2020
63f026f
migrate some more imports
Stebalien Sep 21, 2020
d33dd4f
more renames
Stebalien Sep 21, 2020
d56da1b
Refinements to stmgr and utils
arajasek Sep 22, 2020
1dc69e3
Resolve some unnecessary actor upgrade TODOs
arajasek Sep 22, 2020
fc15888
fixup some imports
Stebalien Sep 22, 2020
5314ba8
remove ptr indirection
Stebalien Sep 22, 2020
ab070f2
copy actor object when iterating over actors
Stebalien Sep 22, 2020
bc24fdb
finish migrating statemanager
Stebalien Sep 22, 2020
441d7ff
cleanup imports some more
Stebalien Sep 22, 2020
c91774b
remove old comment
Stebalien Sep 22, 2020
2967c4e
use abstract actor type methods
Stebalien Sep 22, 2020
46fb0e7
add deal state iterator
Stebalien Sep 22, 2020
77f81fc
Test fix
arajasek Sep 22, 2020
7737147
update oni
Stebalien Sep 23, 2020
ed4bf9b
API shouldn't depend on actors directly
arajasek Sep 23, 2020
e27fc03
Reward state interface only needs cbor.Marshaler
arajasek Sep 23, 2020
476e799
Add an error return to all actor state interface methods
arajasek Sep 23, 2020
819180f
Implement inefficient OnAddressMapChange predicate
arajasek Sep 23, 2020
a876a0b
Use actor state addresses
arajasek Sep 23, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions api/api_full.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/big"
"github.com/filecoin-project/go-state-types/crypto"
"github.com/filecoin-project/go-state-types/dline"
"github.com/filecoin-project/specs-actors/actors/builtin/market"
"github.com/filecoin-project/specs-actors/actors/builtin/miner"
"github.com/filecoin-project/specs-actors/actors/builtin/paych"
Expand Down Expand Up @@ -312,15 +313,11 @@ type FullNode interface {
StateMinerActiveSectors(context.Context, address.Address, types.TipSetKey) ([]*ChainSectorInfo, error)
// StateMinerProvingDeadline calculates the deadline at some epoch for a proving period
// and returns the deadline-related calculations.
StateMinerProvingDeadline(context.Context, address.Address, types.TipSetKey) (*miner.DeadlineInfo, error)
StateMinerProvingDeadline(context.Context, address.Address, types.TipSetKey) (*dline.Info, error)
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
// StateMinerPower returns the power of the indicated miner
StateMinerPower(context.Context, address.Address, types.TipSetKey) (*MinerPower, error)
// StateMinerInfo returns info about the indicated miner
StateMinerInfo(context.Context, address.Address, types.TipSetKey) (MinerInfo, error)
// StateMinerDeadlines returns all the proving deadlines for the given miner
StateMinerDeadlines(context.Context, address.Address, types.TipSetKey) ([]*miner.Deadline, error)
// StateMinerPartitions loads miner partitions for the specified miner/deadline
StateMinerPartitions(context.Context, address.Address, uint64, types.TipSetKey) ([]*miner.Partition, error)
// StateMinerFaults returns a bitfield indicating the faulty sectors of the given miner
StateMinerFaults(context.Context, address.Address, types.TipSetKey) (bitfield.BitField, error)
// StateAllMinerFaults returns all non-expired Faults that occur within lookback epochs of the given tipset
Expand Down
47 changes: 4 additions & 43 deletions api/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ import (
datatransfer "github.com/filecoin-project/go-data-transfer"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/build"
"github.com/filecoin-project/specs-actors/actors/builtin/miner"
"github.com/ipfs/go-cid"

v0miner "github.com/filecoin-project/specs-actors/actors/builtin/miner"
Stebalien marked this conversation as resolved.
Show resolved Hide resolved

"github.com/libp2p/go-libp2p-core/peer"
pubsub "github.com/libp2p/go-libp2p-pubsub"
ma "github.com/multiformats/go-multiaddr"
Expand Down Expand Up @@ -49,48 +50,6 @@ type PubsubScore struct {
Score *pubsub.PeerScoreSnapshot
}

type MinerInfo struct {
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
Owner address.Address // Must be an ID-address.
Worker address.Address // Must be an ID-address.
NewWorker address.Address // Must be an ID-address.
ControlAddresses []address.Address // Must be an ID-addresses.
WorkerChangeEpoch abi.ChainEpoch
PeerId *peer.ID
Multiaddrs []abi.Multiaddrs
SealProofType abi.RegisteredSealProof
SectorSize abi.SectorSize
WindowPoStPartitionSectors uint64
}

func NewApiMinerInfo(info *miner.MinerInfo) MinerInfo {
var pid *peer.ID
if peerID, err := peer.IDFromBytes(info.PeerId); err == nil {
pid = &peerID
}

mi := MinerInfo{
Owner: info.Owner,
Worker: info.Worker,
ControlAddresses: info.ControlAddresses,

NewWorker: address.Undef,
WorkerChangeEpoch: -1,

PeerId: pid,
Multiaddrs: info.Multiaddrs,
SealProofType: info.SealProofType,
SectorSize: info.SectorSize,
WindowPoStPartitionSectors: info.WindowPoStPartitionSectors,
}

if info.PendingWorkerKey != nil {
mi.NewWorker = info.PendingWorkerKey.NewWorker
mi.WorkerChangeEpoch = info.PendingWorkerKey.EffectiveAt
}

return mi
}

type MessageSendSpec struct {
MaxFee abi.TokenAmount
}
Expand Down Expand Up @@ -151,3 +110,5 @@ func NewDataTransferChannel(hostID peer.ID, channelState datatransfer.ChannelSta
}
return channel
}

type
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
57 changes: 57 additions & 0 deletions chain/actors/adt/adt.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package adt

import (
"github.com/ipfs/go-cid"
"golang.org/x/xerrors"

"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/cbor"
"github.com/filecoin-project/go-state-types/network"
"github.com/filecoin-project/lotus/chain/actors/builtin"
v0adt "github.com/filecoin-project/specs-actors/actors/util/adt"
)

type Map interface {
Root() (cid.Cid, error)

Put(k abi.Keyer, v cbor.Marshaler) error
Get(k abi.Keyer, v cbor.Unmarshaler) (bool, error)
Delete(k abi.Keyer) error

ForEach(v cbor.Unmarshaler, fn func(key string) error) error
}

func AsMap(store Store, root cid.Cid, version network.Version) (Map, error) {
switch builtin.VersionForNetwork(version) {
case builtin.Version0:
return v0adt.AsMap(store, root)
}
return nil, xerrors.Errorf("unknown network version: %d", version)
}

func NewMap(store Store, version network.Version) (Map, error) {
switch builtin.VersionForNetwork(version) {
case builtin.Version0:
return v0adt.MakeEmptyMap(store)
}
return nil, xerrors.Errorf("unknown network version: %d", version)
}

type Array interface {
Root() (cid.Cid, error)

Set(idx uint64, v cbor.Marshaler) error
Get(idx uint64, v cbor.Unmarshaler) (bool, error)
Delete(idx uint64) error
Length() uint64

ForEach(v cbor.Unmarshaler, fn func(idx int) error) error
}

func AsArray(store Store, root cid.Cid, version network.Version) (Array, error) {
switch builtin.VersionForNetwork(version) {
case builtin.Version0:
return v0adt.AsArray(store, root)
}
return nil, xerrors.Errorf("unknown network version: %d", version)
}
17 changes: 17 additions & 0 deletions chain/actors/adt/store.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package adt

import (
"context"

adt "github.com/filecoin-project/specs-actors/actors/util/adt"
cbor "github.com/ipfs/go-ipld-cbor"
)

type Store interface {
Context() context.Context
cbor.IpldStore
}

func WrapStore(ctx context.Context, store cbor.IpldStore) Store {
return adt.WrapStore(ctx, store)
}
29 changes: 29 additions & 0 deletions chain/actors/builtin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Actors

This package contains shims for abstracting over different actor versions.

## Design

Shims in this package follow a few common design principles.

### Structure Agnostic

Shims interfaces defined in this package should (ideally) not change even if the
structure of the underlying data changes. For example:

* All shims store an internal "store" object. That way, state can be moved into
a separate object without needing to add a store to the function signature.
* All functions must return an error, even if unused for now.

### Minimal

These interfaces should be expanded only as necessary to reduce maintenance burden.

### Queries, not field assessors.

When possible, functions should query the state instead of simply acting as
field assessors. These queries are more likely to remain stable across
specs-actor upgrades than specific state fields.

Note: there is a trade-off here. Avoid implementing _complicated_ query logic
inside these shims, as it will need to be replicated in every shim.
23 changes: 23 additions & 0 deletions chain/actors/builtin/builtin.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package builtin

import (
"fmt"

"github.com/filecoin-project/go-state-types/network"
)

type Version int

const (
Version0 = iota
)

// Converts a network version into a specs-actors version.
func VersionForNetwork(version network.Version) Version {
switch version {
case network.Version0, network.Version1:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Network version != actor version. Also, actor version is going to go from 0 (well, 0.9) to 2. That's going to be rather strange..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it will just be used as a unique key, maybe use a string "v2" instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're now encoding this inside the state tree itself.

@magik6k what version do we actually want in the state tree. We could use:

  • Network version (seems like overkill).
  • Actors version (what we're using here).
  • A specific "state tree" version (for the HAMT). That seems like the best approach, honestly.

Thoughts?

Copy link
Contributor

@magik6k magik6k Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A specific state tree version sounds like what we want here

return Version
default:
panic(fmt.Sprintf("unsupported network version %d", version))
}
}
32 changes: 32 additions & 0 deletions chain/actors/builtin/init/init.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package init

import (
"github.com/filecoin-project/go-bitfield"
"github.com/filecoin-project/go-state-types/cbor"
"golang.org/x/xerrors"

v0builtin "github.com/filecoin-project/specs-actors/actors/builtin"

"github.com/filecoin-project/lotus/chain/actors/adt"
"github.com/filecoin-project/lotus/chain/types"
)

func Load(store adt.Store, act *types.Actor) (State, error) {
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
switch act.Code {
case v0builtin.InitActorCodeID:
out := v0State{store: store}
err := store.Get(store.Context(), act.Head, &out)
if err != nil {
return nil, err
}
return &out, nil
}
return nil, xerrors.Errorf("unknown actor code %s", act.Code)
}

type State interface {
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
cbor.Marshaler

ResolveAddress(address addr.Address) (address.Address, bool, error)
MapAddressToNewID(address addr.Address) (address.Address, error)
}
22 changes: 22 additions & 0 deletions chain/actors/builtin/init/v0.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package init

import (
"github.com/filecoin-project/go-address"

"github.com/filecoin-project/specs-actors/actors/builtin/init"

"github.com/filecoin-project/lotus/chain/actors/adt"
)

type v0State struct {
init.State
store adt.Store
}

func (s *v0State) ResolveAddress(address address.Address) (address.Address, bool, error) {
return s.State.ResolveAddress(s.store, address)
}

func (s *v0State) MapAddressToNewID(address address.Address) (address.Address, error) {
return s.State.MapAddressToNewID(s.store, address)
}
32 changes: 32 additions & 0 deletions chain/actors/builtin/market/market.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package market

import (
"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/cbor"
"github.com/ipfs/go-cid"
)

func Load(store adt.Store, act *types.Actor) (st State, err error) {
switch act.Code {
case v0builtin.MarketActorCodeID:
out := v0State{store: store}
err := store.Get(store.Context(), act.Head, &out)
if err != nil {
return nil, err
}
return &out, nil
}
return nil, xerrors.Errorf("unknown actor code %s", act.Code)
}

type State interface {
cbor.Marshaler
EscrowTable() (BalanceTable, error)
LockedTable() (BalanceTable, error)
TotalLocked() (abi.TokenAmount, error)
}

type BalanceTable interface {
Get(key address.Address) (abi.TokenAmount, error)
}
25 changes: 25 additions & 0 deletions chain/actors/builtin/market/v0.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package market

import (
"github.com/filecoin-project/specs-actors/actors/builtin/market"
"github.com/filecoin-project/specs-actors/actors/util/adt"
)

type v0State struct {
market.State
store adt.Store
}
Stebalien marked this conversation as resolved.
Show resolved Hide resolved

func (s *v0State) TotalLocked() (abi.TokenAmount, error) {
fml := types.BigAdd(s.TotalClientLockedCollateral, s.TotalProviderLockedCollateral)
fml = types.BigAdd(fml, s.TotalClientStorageFee)
return fml, nil
}

func (s *v0State) EscrowTable() (BalanceTable, error) {
return adt.AsBalanceTable(s.store, s.State.EscrowTable)
}

func (s *v0State) Lockedtable() (BalanceTable, error) {
return adt.AsBalanceTable(s.store, s.State.LockedTable)
}
59 changes: 59 additions & 0 deletions chain/actors/builtin/miner/miner.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package miner

import (
"github.com/filecoin-project/go-bitfield"
"github.com/filecoin-project/go-state-types/abi"
"golang.org/x/xerrors"

v0builtin "github.com/filecoin-project/specs-actors/actors/builtin"

"github.com/filecoin-project/lotus/chain/actors/adt"
"github.com/filecoin-project/lotus/chain/types"
)

func Load(store adt.Store, act *types.Actor) (st State, err error) {
switch act.Code {
case v0builtin.StorageMinerActorCodeID:
out := v0State{store: store}
err := store.Get(store.Context(), act.Head, &out)
if err != nil {
return nil, err
}
return &out, nil
}
return nil, xerrors.Errorf("unknown actor code %s", act.Code)
}

type State interface {
cbor.Marshaler

LoadDeadline(idx uint64) (Deadline, error)
ForEachDeadline(cb func(idx uint64, dl Deadline) error) error
NumDeadlines() (uint64, error)
}

type Deadline interface {
LoadPartition(idx uint64) (Partition, error)
ForEachPartition(cb func(idx uint64, part Partition) error) error
}

type Partition interface {
AllSectors() (bitfield.BitField, error)
FaultySectors() (bitfield.BitField, error)
RecoveringSectors() (bitfield.BitField, error)
LiveSectors() (bitfield.BitField, error)
ActiveSectors() (bitfield.BitField, error)
}

type MinerInfo struct {
Owner address.Address // Must be an ID-address.
Worker address.Address // Must be an ID-address.
NewWorker address.Address // Must be an ID-address.
ControlAddresses []address.Address // Must be an ID-addresses.
WorkerChangeEpoch abi.ChainEpoch
PeerId *peer.ID
Multiaddrs []abi.Multiaddrs
SealProofType abi.RegisteredSealProof
SectorSize abi.SectorSize
WindowPoStPartitionSectors uint64
}
Loading