Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network upgrade support #3781

Merged
merged 110 commits into from
Sep 23, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
d359483
[WIP] Network upgrade support
Stebalien Sep 12, 2020
38f8798
Fix some build errors
magik6k Sep 14, 2020
6809713
Fix vm build
magik6k Sep 14, 2020
683a581
More terraforming in chain/
magik6k Sep 14, 2020
06ec571
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
magik6k Sep 14, 2020
fb3f710
Some post-master merge fixes
magik6k Sep 14, 2020
2088335
rename to simplify merge
Stebalien Sep 14, 2020
02dcb5e
Merge branch 'master' into refactor/net-upgrade
Stebalien Sep 14, 2020
cc4d530
Progress
Stebalien Sep 14, 2020
7fa6c1e
add info method to miner state API
Stebalien Sep 14, 2020
36f920b
progress
Stebalien Sep 15, 2020
4dabab5
state manager progress
magik6k Sep 15, 2020
4419850
state api impl fixes
magik6k Sep 15, 2020
9dad38c
gofmt
magik6k Sep 15, 2020
00c6397
more progress
Stebalien Sep 15, 2020
6bf7976
fix decode
Stebalien Sep 15, 2020
4cd92d8
remove final WithStateTree
Stebalien Sep 15, 2020
53384e8
migrate wallet access
Stebalien Sep 15, 2020
c64f983
migrate StateMinserSectorCount
Stebalien Sep 15, 2020
91e9573
Compile fixes
Stebalien Sep 15, 2020
4e01fad
start multisig abstraction
Stebalien Sep 15, 2020
e1ba4eb
sub reward actor
Stebalien Sep 15, 2020
02bc5fa
add power actor address alias
Stebalien Sep 15, 2020
92471d4
migrate precommit deposit function
Stebalien Sep 15, 2020
454c382
migrate StateMinerInitialPledgeCollateral
Stebalien Sep 16, 2020
05c1153
feat(paych): convert paych actor
hannahhoward Sep 16, 2020
b4ee519
Partial progress towards switching to miner and power interfaces
arajasek Sep 16, 2020
90853e2
Add a boolean HasMinPower to return of GetPower
arajasek Sep 16, 2020
72d19f3
Incremental progress towards using new power state interface
arajasek Sep 16, 2020
ccce1a9
Merge pull request #3877 from filecoin-project/refactor/net-upgrade-p…
magik6k Sep 16, 2020
b530f25
Migrate miner actor
arajasek Sep 16, 2020
7115485
Add a getter for network version
arajasek Sep 17, 2020
80b6994
feat(market): update state diffing for market actor
hannahhoward Sep 17, 2020
81004ea
Merge pull request #3889 from filecoin-project/refactor/net-upgrade-m…
arajasek Sep 17, 2020
b5ba7a0
Miner migration
arajasek Sep 17, 2020
9e48dd2
Fixups
arajasek Sep 17, 2020
b606149
Migrate reward actor
arajasek Sep 17, 2020
e2295c3
Migrate multisig actor
arajasek Sep 17, 2020
691bd9f
feat(markets): complete markets conversion
hannahhoward Sep 17, 2020
7bf165c
Remove miner-related specs actors types from API
arajasek Sep 17, 2020
053cfc1
Migrate verifreg actor
arajasek Sep 17, 2020
31ff523
Get State API almost working
arajasek Sep 17, 2020
e632643
api: Test return types
magik6k Sep 17, 2020
6eda535
Most tests passing
magik6k Sep 17, 2020
68e884e
docsgen
magik6k Sep 17, 2020
82b95e3
cbor gen
magik6k Sep 17, 2020
c40c136
fix: paych To()
dirkmc Sep 17, 2020
18bc59b
uncruft
Stebalien Sep 17, 2020
708aa36
remove dep on v0 miner
Stebalien Sep 17, 2020
ae38970
remove WpostProvignPeriod function
Stebalien Sep 17, 2020
dc58f71
remove unnecessary code
Stebalien Sep 17, 2020
5bcfee0
make market diffs work across version upgrades
Stebalien Sep 17, 2020
b2ee590
improve diff logic
Stebalien Sep 18, 2020
f2a0779
remove specs-actors import
Stebalien Sep 18, 2020
8bd6791
more fixups
Stebalien Sep 18, 2020
daa441b
simplify market diff
Stebalien Sep 18, 2020
37de154
Fixup tests
arajasek Sep 18, 2020
fce423c
Appease the linter and get everything building
arajasek Sep 18, 2020
694463f
More lint fixes
arajasek Sep 18, 2020
70faa36
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
magik6k Sep 18, 2020
ccd0a67
wip fixing wdpost tests
magik6k Sep 18, 2020
c3046f4
Don't use -1 as default sector number
arajasek Sep 18, 2020
fb2b25c
finish upgrading chainwatch
Stebalien Sep 18, 2020
298efa2
fix lint errors
Stebalien Sep 18, 2020
8747c60
abstract over account actor
Stebalien Sep 18, 2020
24df873
rename imports
Stebalien Sep 18, 2020
e60027c
remove todo
Stebalien Sep 18, 2020
8285eda
migrate storage miner info
Stebalien Sep 18, 2020
4a7055c
fix state loading in vm
Stebalien Sep 18, 2020
f741ce6
fixup some more imports
Stebalien Sep 18, 2020
c130806
compile fix
Stebalien Sep 18, 2020
1bf3b49
rename imports to match actors code
Stebalien Sep 18, 2020
35bce5a
revert post changes
Stebalien Sep 18, 2020
7c3f638
Abstract FilterEstimate, PreCommitDepositForPower, and InitialPledgeF…
arajasek Sep 19, 2020
b25dd2a
More correct / fasterer GetSectorsForWinningPoSt
magik6k Sep 20, 2020
ed285f1
Abstract SectorOnChainInfo and SectorPreCommitOnChainInfo
arajasek Sep 20, 2020
a95e34f
Fix build
arajasek Sep 21, 2020
b355eb7
Add compile-time checks that actors interfaces are correctly implemented
arajasek Sep 21, 2020
286fe04
Add some TODOs
arajasek Sep 21, 2020
a307e45
wdpost: Fix TestWDPostDoPost
magik6k Sep 21, 2020
0b5e4a9
Make GetSectorsForWinningPoSt fast again
magik6k Sep 21, 2020
f7934b0
Merge remote-tracking branch 'origin/master' into refactor/net-upgrade
magik6k Sep 21, 2020
0256631
non-destructive diff
Stebalien Sep 21, 2020
4cf0c10
optimize sector loading
Stebalien Sep 21, 2020
24ae920
cross-version state tree diff
Stebalien Sep 21, 2020
f9a896f
fix compile
Stebalien Sep 21, 2020
0ab2459
convert multisig to new actor abstractions
Stebalien Sep 21, 2020
a41bf74
fix: remove incorrect variable
Stebalien Sep 21, 2020
916421b
convert lotus-shed balances
Stebalien Sep 21, 2020
4bab784
migrate lotus-shed verifreg to specs-actors abstractions
Stebalien Sep 21, 2020
3f0106c
migrate lotus-shed/genesis-verify to actor abstraction
Stebalien Sep 21, 2020
63f026f
migrate some more imports
Stebalien Sep 21, 2020
d33dd4f
more renames
Stebalien Sep 21, 2020
d56da1b
Refinements to stmgr and utils
arajasek Sep 22, 2020
1dc69e3
Resolve some unnecessary actor upgrade TODOs
arajasek Sep 22, 2020
fc15888
fixup some imports
Stebalien Sep 22, 2020
5314ba8
remove ptr indirection
Stebalien Sep 22, 2020
ab070f2
copy actor object when iterating over actors
Stebalien Sep 22, 2020
bc24fdb
finish migrating statemanager
Stebalien Sep 22, 2020
441d7ff
cleanup imports some more
Stebalien Sep 22, 2020
c91774b
remove old comment
Stebalien Sep 22, 2020
2967c4e
use abstract actor type methods
Stebalien Sep 22, 2020
46fb0e7
add deal state iterator
Stebalien Sep 22, 2020
77f81fc
Test fix
arajasek Sep 22, 2020
7737147
update oni
Stebalien Sep 23, 2020
ed4bf9b
API shouldn't depend on actors directly
arajasek Sep 23, 2020
e27fc03
Reward state interface only needs cbor.Marshaler
arajasek Sep 23, 2020
476e799
Add an error return to all actor state interface methods
arajasek Sep 23, 2020
819180f
Implement inefficient OnAddressMapChange predicate
arajasek Sep 23, 2020
a876a0b
Use actor state addresses
arajasek Sep 23, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 2 additions & 7 deletions api/api_full.go
Original file line number Diff line number Diff line change
Expand Up @@ -315,9 +315,9 @@ type FullNode interface {
// StateMinerSectors returns info about the given miner's sectors. If the filter bitfield is nil, all sectors are included.
// If the filterOut boolean is set to true, any sectors in the filter are excluded.
// If false, only those sectors in the filter are included.
StateMinerSectors(context.Context, address.Address, *bitfield.BitField, bool, types.TipSetKey) ([]*ChainSectorInfo, error)
StateMinerSectors(context.Context, address.Address, *bitfield.BitField, bool, types.TipSetKey) ([]*miner2.ChainSectorInfo, error)
// StateMinerActiveSectors returns info about sectors that a given miner is actively proving.
StateMinerActiveSectors(context.Context, address.Address, types.TipSetKey) ([]*ChainSectorInfo, error)
StateMinerActiveSectors(context.Context, address.Address, types.TipSetKey) ([]*miner2.ChainSectorInfo, error)
// StateMinerProvingDeadline calculates the deadline at some epoch for a proving period
// and returns the deadline-related calculations.
StateMinerProvingDeadline(context.Context, address.Address, types.TipSetKey) (*dline.Info, error)
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -538,11 +538,6 @@ type Message struct {
Message *types.Message
}

type ChainSectorInfo struct {
Info miner.SectorOnChainInfo
ID abi.SectorNumber
}

type ActorState struct {
Balance types.BigInt
State interface{}
Expand Down
30 changes: 15 additions & 15 deletions api/apistruct/struct.go
Original file line number Diff line number Diff line change
Expand Up @@ -161,19 +161,19 @@ type FullNodeStruct struct {
ClientDataTransferUpdates func(ctx context.Context) (<-chan api.DataTransferChannel, error) `perm:"write"`
ClientRetrieveTryRestartInsufficientFunds func(ctx context.Context, paymentChannel address.Address) error `perm:"write"`

StateNetworkName func(context.Context) (dtypes.NetworkName, error) `perm:"read"`
StateMinerSectors func(context.Context, address.Address, *bitfield.BitField, bool, types.TipSetKey) ([]*api.ChainSectorInfo, error) `perm:"read"`
StateMinerActiveSectors func(context.Context, address.Address, types.TipSetKey) ([]*api.ChainSectorInfo, error) `perm:"read"`
StateMinerProvingDeadline func(context.Context, address.Address, types.TipSetKey) (*dline.Info, error) `perm:"read"`
StateMinerPower func(context.Context, address.Address, types.TipSetKey) (*api.MinerPower, error) `perm:"read"`
StateMinerInfo func(context.Context, address.Address, types.TipSetKey) (miner2.MinerInfo, error) `perm:"read"`
StateMinerFaults func(context.Context, address.Address, types.TipSetKey) (bitfield.BitField, error) `perm:"read"`
StateAllMinerFaults func(context.Context, abi.ChainEpoch, types.TipSetKey) ([]*api.Fault, error) `perm:"read"`
StateMinerRecoveries func(context.Context, address.Address, types.TipSetKey) (bitfield.BitField, error) `perm:"read"`
StateMinerPreCommitDepositForPower func(context.Context, address.Address, miner.SectorPreCommitInfo, types.TipSetKey) (types.BigInt, error) `perm:"read"`
StateMinerInitialPledgeCollateral func(context.Context, address.Address, miner.SectorPreCommitInfo, types.TipSetKey) (types.BigInt, error) `perm:"read"`
StateMinerAvailableBalance func(context.Context, address.Address, types.TipSetKey) (types.BigInt, error) `perm:"read"`
StateSectorPreCommitInfo func(context.Context, address.Address, abi.SectorNumber, types.TipSetKey) (miner.SectorPreCommitOnChainInfo, error) `perm:"read"`
StateNetworkName func(context.Context) (dtypes.NetworkName, error) `perm:"read"`
StateMinerSectors func(context.Context, address.Address, *bitfield.BitField, bool, types.TipSetKey) ([]*miner2.ChainSectorInfo, error) `perm:"read"`
StateMinerActiveSectors func(context.Context, address.Address, types.TipSetKey) ([]*miner2.ChainSectorInfo, error) `perm:"read"`
StateMinerProvingDeadline func(context.Context, address.Address, types.TipSetKey) (*dline.Info, error) `perm:"read"`
StateMinerPower func(context.Context, address.Address, types.TipSetKey) (*api.MinerPower, error) `perm:"read"`
StateMinerInfo func(context.Context, address.Address, types.TipSetKey) (miner2.MinerInfo, error) `perm:"read"`
StateMinerFaults func(context.Context, address.Address, types.TipSetKey) (bitfield.BitField, error) `perm:"read"`
StateAllMinerFaults func(context.Context, abi.ChainEpoch, types.TipSetKey) ([]*api.Fault, error) `perm:"read"`
StateMinerRecoveries func(context.Context, address.Address, types.TipSetKey) (bitfield.BitField, error) `perm:"read"`
StateMinerPreCommitDepositForPower func(context.Context, address.Address, miner.SectorPreCommitInfo, types.TipSetKey) (types.BigInt, error) `perm:"read"`
StateMinerInitialPledgeCollateral func(context.Context, address.Address, miner.SectorPreCommitInfo, types.TipSetKey) (types.BigInt, error) `perm:"read"`
StateMinerAvailableBalance func(context.Context, address.Address, types.TipSetKey) (types.BigInt, error) `perm:"read"`
StateSectorPreCommitInfo func(context.Context, address.Address, abi.SectorNumber, types.TipSetKey) (miner.SectorPreCommitOnChainInfo, error) `perm:"read"`
StateSectorGetInfo func(context.Context, address.Address, abi.SectorNumber, types.TipSetKey) (*miner.SectorOnChainInfo, error) `perm:"read"`
StateSectorExpiration func(context.Context, address.Address, abi.SectorNumber, types.TipSetKey) (*api.SectorExpiration, error) `perm:"read"`
StateSectorPartition func(context.Context, address.Address, abi.SectorNumber, types.TipSetKey) (*api.SectorLocation, error) `perm:"read"`
Expand Down Expand Up @@ -730,11 +730,11 @@ func (c *FullNodeStruct) StateNetworkName(ctx context.Context) (dtypes.NetworkNa
return c.Internal.StateNetworkName(ctx)
}

func (c *FullNodeStruct) StateMinerSectors(ctx context.Context, addr address.Address, filter *bitfield.BitField, filterOut bool, tsk types.TipSetKey) ([]*api.ChainSectorInfo, error) {
func (c *FullNodeStruct) StateMinerSectors(ctx context.Context, addr address.Address, filter *bitfield.BitField, filterOut bool, tsk types.TipSetKey) ([]*miner2.ChainSectorInfo, error) {
return c.Internal.StateMinerSectors(ctx, addr, filter, filterOut, tsk)
}

func (c *FullNodeStruct) StateMinerActiveSectors(ctx context.Context, addr address.Address, tsk types.TipSetKey) ([]*api.ChainSectorInfo, error) {
func (c *FullNodeStruct) StateMinerActiveSectors(ctx context.Context, addr address.Address, tsk types.TipSetKey) ([]*miner2.ChainSectorInfo, error) {
return c.Internal.StateMinerActiveSectors(ctx, addr, tsk)
}

Expand Down
6 changes: 6 additions & 0 deletions chain/actors/builtin/miner/miner.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ type State interface {

GetSector(abi.SectorNumber) (*SectorOnChainInfo, error)
GetPrecommittedSector(abi.SectorNumber) (*SectorPreCommitOnChainInfo, error)
LoadSectorsFromSet(filter *bitfield.BitField, filterOut bool) ([]*ChainSectorInfo, error)

LoadDeadline(idx uint64) (Deadline, error)
ForEachDeadline(cb func(idx uint64, dl Deadline) error) error
Expand Down Expand Up @@ -71,3 +72,8 @@ type MinerInfo struct {
SectorSize abi.SectorSize
WindowPoStPartitionSectors uint64
}

type ChainSectorInfo struct {
Info SectorOnChainInfo
ID abi.SectorNumber
}
28 changes: 28 additions & 0 deletions chain/actors/builtin/miner/utils.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package miner

import (
"golang.org/x/xerrors"

"github.com/filecoin-project/go-bitfield"
)

func AllPartSectors(mas State, sget func(Partition) (bitfield.BitField, error)) (bitfield.BitField, error) {
var parts []bitfield.BitField

err := mas.ForEachDeadline(func(dlidx uint64, dl Deadline) error {
return dl.ForEachPartition(func(partidx uint64, part Partition) error {
s, err := sget(part)
if err != nil {
return xerrors.Errorf("getting sector list (dl: %d, part %d): %w", dlidx, partidx, err)
}

parts = append(parts, s)
return nil
})
})
if err != nil {
return bitfield.BitField{}, err
}

return bitfield.MultiMerge(parts...)
}
39 changes: 39 additions & 0 deletions chain/actors/builtin/miner/v0.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@ import (
"github.com/filecoin-project/go-bitfield"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/chain/actors/adt"
v0adt "github.com/filecoin-project/specs-actors/actors/util/adt"
cbor "github.com/ipfs/go-ipld-cbor"
"github.com/libp2p/go-libp2p-core/peer"
cbg "github.com/whyrusleeping/cbor-gen"
"golang.org/x/xerrors"

"github.com/filecoin-project/specs-actors/actors/builtin/miner"
)
Expand Down Expand Up @@ -43,6 +47,41 @@ func (s *v0State) GetPrecommittedSector(num abi.SectorNumber) (*SectorPreCommitO
return info, nil
}

func (s *v0State) LoadSectorsFromSet(filter *bitfield.BitField, filterOut bool) ([]*ChainSectorInfo, error) {
a, err := v0adt.AsArray(s.store, s.State.Sectors)
if err != nil {
return nil, err
}

var sset []*ChainSectorInfo
var v cbg.Deferred
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@magik6k urgh, just realised that this undoes the fasterer changes you introduced here: b25dd2a.

But we do need to do some dance manipulating these, since the SectorOnChainInfo changes between actors 0 and 1. Kick the performance implications down the road for now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't, this takes 5min+ for bigger miners, and we call this on each epoch, multiple times

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have rewritten this to be faster. It should now fetch exactly the sectors requested.

if err := a.ForEach(&v, func(i int64) error {
if filter != nil {
set, err := filter.IsSet(uint64(i))
if err != nil {
return xerrors.Errorf("filter check error: %w", err)
}
if set == filterOut {
return nil
}
}

var oci miner.SectorOnChainInfo
if err := cbor.DecodeInto(v.Raw, &oci); err != nil {
return err
}
sset = append(sset, &ChainSectorInfo{
Info: oci,
ID: abi.SectorNumber(i),
})
return nil
}); err != nil {
return nil, err
}

return sset, nil
}

func (s *v0State) LoadDeadline(idx uint64) (Deadline, error) {
dls, err := s.State.LoadDeadlines(s.store)
if err != nil {
Expand Down
Loading