Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: events: Lotus API to consume events (smart contract + built-in Actor events) #11540

Closed
wants to merge 76 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
8db6e73
Bump version in master
rjan90 Dec 4, 2023
c9f1325
Make gen
rjan90 Dec 5, 2023
271a011
Merge pull request #11475 from filecoin-project/phi-bump-master-v
magik6k Dec 5, 2023
538039f
fix sender
snadrus Dec 5, 2023
956ac8a
Merge pull request #11480 from filecoin-project/lpSenderBug
magik6k Dec 5, 2023
943c223
fix: lotus-provider: Fix winning PoSt
magik6k Dec 5, 2023
2d75cc5
Merge pull request #11482 from filecoin-project/fix/lp-winning
snadrus Dec 5, 2023
685934a
fix: sql Scan cannot write to an object
snadrus Dec 5, 2023
91b0032
do not suggest the default layer
snadrus Dec 5, 2023
e0a8fe3
Merge pull request #11485 from filecoin-project/sqlGptHard
magik6k Dec 5, 2023
e570693
lp docsgen
snadrus Dec 5, 2023
3d30ded
Actually show miner-addrs in info-log
rjan90 Dec 6, 2023
9dd9458
Merge pull request #11490 from filecoin-project/phi-fix-lp-info-log
snadrus Dec 6, 2023
4b1445e
fix: lotus-provider: Wait for the correct taskID
magik6k Dec 6, 2023
93fd408
lotus-provider: additional fixes to make recover work
magik6k Dec 6, 2023
bf5132e
more sender fixes
magik6k Dec 6, 2023
d32b8be
Merge pull request #11493 from filecoin-project/fix/lp-send-taskid-wait
snadrus Dec 6, 2023
efb4a09
harmony: Fix task reclaim on restart
magik6k Dec 7, 2023
cf8fed9
Merge pull request #11498 from filecoin-project/fix/harmony-reclaim
snadrus Dec 7, 2023
0e49673
fix: db serialize txn - retry
snadrus Dec 7, 2023
1e09e1e
detect unsafe code uses
snadrus Dec 7, 2023
4ab92ca
Fix log output format in wdPostTaskCmd
rjan90 Dec 8, 2023
986f240
fix: exchange: allow up to 10k messages per block
Stebalien Dec 9, 2023
88e90aa
fix lints
Stebalien Dec 9, 2023
7f684ec
Merge pull request #11506 from filecoin-project/steb/fix-exchange-mes…
magik6k Dec 9, 2023
f2a4891
fix: exchange: explicitly cast the block message limit const (#11511)
Stebalien Dec 9, 2023
4a5c241
bump version
jennijuju Dec 9, 2023
9dca434
fix: lint
snadrus Dec 9, 2023
68c1e56
Merge pull request #11504 from filecoin-project/lp-wdposttaskcmd-format
magik6k Dec 11, 2023
89fabbb
Merge pull request #11488 from filecoin-project/lpdocsgen
magik6k Dec 11, 2023
47477c0
Merge pull request #11486 from filecoin-project/lpMigrateNicer
magik6k Dec 11, 2023
96353e6
exp backoff, short stack err
snadrus Dec 11, 2023
de38e77
tx-detector use atomic
snadrus Dec 11, 2023
c912f3b
Merge pull request #11501 from filecoin-project/serializationFix
snadrus Dec 11, 2023
6faeaa7
update libp2p deps
aarshkshah1992 Dec 14, 2023
d75d7c7
Merge pull request #11522 from filecoin-project/feat/update-libp2p-deps
magik6k Dec 14, 2023
17f6f4c
fix: eth: decode as actor creation iff "to" is the EAM (#11520)
Stebalien Dec 14, 2023
fac1d11
Merge pull request #11512 from filecoin-project/jen/253
rjan90 Dec 15, 2023
d21802f
first draft with historical API
aarshkshah1992 Dec 19, 2023
cfd57cb
changes to first draft
aarshkshah1992 Dec 19, 2023
f7a36f6
fix tests and docsgen
aarshkshah1992 Dec 19, 2023
9f671af
refactor event filter manager
aarshkshah1992 Dec 19, 2023
55ed4cd
susbcribe API first draft
aarshkshah1992 Dec 19, 2023
4a59c01
check error value of removing filter
aarshkshah1992 Dec 19, 2023
5a0c2bc
itests support the new API
aarshkshah1992 Dec 20, 2023
0bf15fa
fix nits
aarshkshah1992 Dec 20, 2023
874b8ca
test and lint
aarshkshah1992 Dec 20, 2023
f33e154
remove test in CI
aarshkshah1992 Dec 20, 2023
c67ed4b
docsgen
aarshkshah1992 Dec 20, 2023
4d72fda
revert sed change
aarshkshah1992 Dec 20, 2023
3ad99bf
Merge remote-tracking branch 'origin/feat/nv22' into feat/built-in-ac…
aarshkshah1992 Jan 25, 2024
462e061
revert change
aarshkshah1992 Jan 25, 2024
a5fb594
rebase
aarshkshah1992 Jan 25, 2024
db0d736
Merge remote-tracking branch 'origin/feat/nv22' into feat/built-in-ac…
aarshkshah1992 Jan 25, 2024
a9c8959
make gen
aarshkshah1992 Jan 25, 2024
7b6d763
test: DDO onboarding non-market verified data
rvagg Jan 30, 2024
b8f7520
docs and more tests
aarshkshah1992 Jan 30, 2024
b02338a
tests
aarshkshah1992 Jan 30, 2024
a4f89a7
Apply suggestions from code review
aarshkshah1992 Jan 31, 2024
147c47e
Apply suggestions from code review
aarshkshah1992 Jan 31, 2024
2a219e2
lint
aarshkshah1992 Jan 31, 2024
ff17d7f
TEMP: extract events
rvagg Feb 1, 2024
b54d111
Extract buildActorEventsFromMessages and separate event printing code
rvagg Feb 2, 2024
3f2c3a3
solve actor resolution bug
aarshkshah1992 Feb 5, 2024
cc04504
Merge remote-tracking branch 'origin/feat/built-in-actor-events-api' …
aarshkshah1992 Feb 6, 2024
a694690
tests for events API
aarshkshah1992 Feb 7, 2024
f9c891b
lint
aarshkshah1992 Feb 7, 2024
b5fba58
Apply suggestions from code review
aarshkshah1992 Feb 7, 2024
4cfd4f0
Merge pull request #11603 from filecoin-project/rvagg/ddo-verified
aarshkshah1992 Feb 7, 2024
8e71abf
review
aarshkshah1992 Feb 7, 2024
3faac88
Merge remote-tracking branch 'origin/feat/nv22' into feat/built-in-ac…
rvagg Feb 8, 2024
e500393
Clean up DDO+Events tests, add lots of explainer comments
rvagg Feb 8, 2024
1744741
Minor tweaks to events types
rvagg Feb 8, 2024
1bd7257
Merge pull request #11613 from filecoin-project/rvagg/actor-events-tweak
aarshkshah1992 Feb 8, 2024
1880835
make gen
aarshkshah1992 Feb 8, 2024
55784cf
Merge remote-tracking branch 'origin/feat/nv22' into feat/built-in-ac…
rvagg Feb 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions api/api_full.go
Original file line number Diff line number Diff line change
Expand Up @@ -885,6 +885,20 @@ type FullNode interface {

RaftState(ctx context.Context) (*RaftStateData, error) //perm:read
RaftLeader(ctx context.Context) (peer.ID, error) //perm:read

// Actor events

// GetActorEvents returns all FVM and built-in Actor events that match the given filter.
// This is a request/response API.
GetActorEvents(ctx context.Context, filter *types.ActorEventFilter) ([]*types.ActorEvent, error) //perm:read

// SubscribeActorEvents returns a long-lived stream of all FVM and built-in Actor events that match the given filter.
// Events that match the given filter are written to the stream in real-time as they are emitted from the FVM.
// The response stream is closed when the client disconnects or if there is an error while writing an event to the stream.
// This API also allows clients to read all historical events matching the given filter before
// any real-time events are written to the response stream.
// NOTE: THIS API IS ONLY SUPPORTED OVER WEBSOCKETS FOR NOW
SubscribeActorEvents(ctx context.Context, filter *types.SubActorEventFilter) (<-chan *types.ActorEvent, error) //perm:read
}

// reverse interface to the client, called after EthSubscribe
Expand Down
3 changes: 3 additions & 0 deletions api/api_gateway.go
Original file line number Diff line number Diff line change
Expand Up @@ -129,4 +129,7 @@ type Gateway interface {
Web3ClientVersion(ctx context.Context) (string, error)
EthTraceBlock(ctx context.Context, blkNum string) ([]*ethtypes.EthTraceBlock, error)
EthTraceReplayBlockTransactions(ctx context.Context, blkNum string, traceTypes []string) ([]*ethtypes.EthTraceReplayBlockTransaction, error)

GetActorEvents(ctx context.Context, filter *types.ActorEventFilter) ([]*types.ActorEvent, error)
SubscribeActorEvents(ctx context.Context, filter *types.SubActorEventFilter) (<-chan *types.ActorEvent, error)
}
36 changes: 36 additions & 0 deletions api/docgen/docgen.go
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,42 @@ func init() {
VerifiedAllocationKey: nil,
Notify: nil,
})

addExample(&types.ActorEventBlock{
Codec: 0x51,
Value: []byte("data"),
})

addExample(&types.ActorEventFilter{
Addresses: []address.Address{addr},
Fields: map[string][]types.ActorEventBlock{
"abc": {
{
Codec: 0x51,
Value: []byte("ddata"),
},
},
Comment on lines +426 to +431
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced this is a great API; It feels really clunky to have to specify the codec here, which is encouraged by the clunky nature of the events themselves of course (!).

We do have the option of leaving the Codec off and just keeping it map[string][][]byte and matching on that, the bytes are going to remain the same, identical to the Value they provide here. It will lack the precision of saying "this value, which is this codec", but that's not too much different from just "this value".

It's not like we're doing anything fancy with the Codec here of course—e.g. decoding their Value and re-encoding it to the codec specified by the event, the Value is the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rvagg

Yeah this is true. I think just a nice way for clients to ensure they're using the right codec as we use different ones for smart contract vs built-in Actor events so they're not left wondering why their filter isn't matching anything. I think best to keep it for now to make it explicit. We can remove this later once our experimental API gets some feedback.

},
FromEpoch: "earliest",
ToEpoch: "latest",
})

addExample(&types.SubActorEventFilter{
Filter: types.ActorEventFilter{
Addresses: []address.Address{addr},
Fields: map[string][]types.ActorEventBlock{
"abc": {
{
Codec: 0x51,
Value: []byte("data"),
},
},
},
FromEpoch: "earliest",
ToEpoch: "latest",
},
Prefill: true,
})
}

func GetAPIType(name, pkg string) (i interface{}, t reflect.Type, permStruct []reflect.Type) {
Expand Down
30 changes: 30 additions & 0 deletions api/mocks/mock_full.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

52 changes: 52 additions & 0 deletions api/proxy_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified build/openrpc/full.json.gz
Binary file not shown.
Binary file modified build/openrpc/gateway.json.gz
Binary file not shown.
Binary file modified build/openrpc/miner.json.gz
Binary file not shown.
Binary file modified build/openrpc/worker.json.gz
Binary file not shown.
42 changes: 22 additions & 20 deletions chain/events/filter/event.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,14 @@ func isIndexedValue(b uint8) bool {
}

type EventFilter struct {
id types.FilterID
minHeight abi.ChainEpoch // minimum epoch to apply filter or -1 if no minimum
maxHeight abi.ChainEpoch // maximum epoch to apply filter or -1 if no maximum
tipsetCid cid.Cid
addresses []address.Address // list of f4 actor addresses that are extpected to emit the event
keys map[string][][]byte // map of key names to a list of alternate values that may match
maxResults int // maximum number of results to collect, 0 is unlimited
id types.FilterID
minHeight abi.ChainEpoch // minimum epoch to apply filter or -1 if no minimum
maxHeight abi.ChainEpoch // maximum epoch to apply filter or -1 if no maximum
tipsetCid cid.Cid
addresses []address.Address // list of f4 actor addresses that are extpected to emit the event

keysWithCodec map[string][]types.ActorEventBlock // map of key names to a list of alternate values that may match
maxResults int // maximum number of results to collect, 0 is unlimited

mu sync.Mutex
collected []*CollectedEvent
Expand Down Expand Up @@ -194,7 +195,7 @@ func (f *EventFilter) matchAddress(o address.Address) bool {
}

func (f *EventFilter) matchKeys(ees []types.EventEntry) bool {
if len(f.keys) == 0 {
if len(f.keysWithCodec) == 0 {
return true
}
// TODO: optimize this naive algorithm
Expand All @@ -216,19 +217,19 @@ func (f *EventFilter) matchKeys(ees []types.EventEntry) bool {
continue
}

wantlist, ok := f.keys[keyname]
wantlist, ok := f.keysWithCodec[keyname]
if !ok || len(wantlist) == 0 {
continue
}

for _, w := range wantlist {
if bytes.Equal(w, ee.Value) {
if bytes.Equal(w.Value, ee.Value) && w.Codec == ee.Codec {
matched[keyname] = true
break
}
}

if len(matched) == len(f.keys) {
if len(matched) == len(f.keysWithCodec) {
// all keys have been matched
return true
}
Expand Down Expand Up @@ -362,7 +363,8 @@ func (m *EventFilterManager) Revert(ctx context.Context, from, to *types.TipSet)
return nil
}

func (m *EventFilterManager) Install(ctx context.Context, minHeight, maxHeight abi.ChainEpoch, tipsetCid cid.Cid, addresses []address.Address, keys map[string][][]byte) (*EventFilter, error) {
func (m *EventFilterManager) Install(ctx context.Context, minHeight, maxHeight abi.ChainEpoch, tipsetCid cid.Cid, addresses []address.Address,
keysWithCodec map[string][]types.ActorEventBlock, excludeReverted bool) (*EventFilter, error) {
m.mu.Lock()
currentHeight := m.currentHeight
m.mu.Unlock()
Expand All @@ -377,18 +379,18 @@ func (m *EventFilterManager) Install(ctx context.Context, minHeight, maxHeight a
}

f := &EventFilter{
id: id,
minHeight: minHeight,
maxHeight: maxHeight,
tipsetCid: tipsetCid,
addresses: addresses,
keys: keys,
maxResults: m.MaxFilterResults,
id: id,
minHeight: minHeight,
maxHeight: maxHeight,
tipsetCid: tipsetCid,
addresses: addresses,
keysWithCodec: keysWithCodec,
maxResults: m.MaxFilterResults,
}

if m.EventIndex != nil && minHeight != -1 && minHeight < currentHeight {
// Filter needs historic events
if err := m.EventIndex.PrefillFilter(ctx, f, true); err != nil {
if err := m.EventIndex.PrefillFilter(ctx, f, excludeReverted); err != nil {
return nil, err
}
}
Expand Down
Loading