Log event provider #9583

amirylm · 2023-06-13T19:21:52Z

This PR introduce LogEventProvider for managing filters registration and query the log poller for logs.

Changes includes:

acting upon canceled / paused upkeep events to mark as inactive and unregister logs
register log filter upon registration or config changes
expose interface for log trigger pre-processor to get logs data
configurable timing, sizes, rates and partitioning options
background fetching of logs by the provider
- rate limiting of blocks
- dynamic range based on last poll block per upkeep
- partitioning of upkeeps to balance the load, by hash based on contract address.
log event buffer - a cache made of a ring buffer of blocks, where each block holds the corresponding logs

github-actions · 2023-06-13T19:22:19Z

I see that you haven't updated any CHANGELOG files. Would it make sense to do so?

ferglor · 2023-06-14T12:33:25Z

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go

+const (
+	// logRetention is the amount of time to retain logs for.
+	// 5 minutes is the desired retention time for logs, but we add an extra 10 minutes buffer.
+	logRetention            = (time.Minute * 5) + (time.Minute * 10)


🏄 For time values I think having the number precede the unit is more legible, e.g. 5 * time.Minute

ferglor · 2023-06-14T12:35:16Z

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go

+}
+
+// Register creates a filter from the given upkeep and calls log poller to register it.
+func (lfm *logEventProvider) RegisterFilter(upkeepID *big.Int, cfg LogTriggerConfig) error {


🏄 I prefer single-letter receiver identifiers, in this instance I'd go for p

ferglor · 2023-06-14T12:36:24Z

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go

+
+// Unregister removes the filter for the given upkeepID
+func (lfm *logEventProvider) UnregisterFilter(upkeepID *big.Int) error {
+	err := lfm.poller.UnregisterFilter(lfm.filterName(upkeepID), nil)


if err := lfm.poller.UnregisterFilter...

reductionista · 2023-06-14T17:14:19Z

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go

+	}
+}
+
+// Register creates a filter from the given upkeep and calls log poller to register it.


Minor docstring typo: should say "RegisterFilter creates... "

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go

reductionista · 2023-06-15T18:26:33Z

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go

+		filtered := make([]logpoller.Log, 0)
+		for _, log := range logs {
+			if entry.lastPollBlock > log.BlockNumber {
+				// TODO: check if the log is really known and not a result of some reorg


If you pass chain.Config().EvmFinalityDepth() into the constructor for logEventProvider, and save it somewhere, you can use it to either get only finalized logs back from log poller... or to check afterwards whether it's safe to assume a log can no longer be re-orged.

You can either use a query that accepts minConfs, passing minConfs=finalityDepth or set the end block range to latest - finalityDepth

FelixFan1992

should this be a part of keepers/2_1?

core/services/ocr2/plugins/ocr2keeper/integration_test.go

core/services/ocr2/plugins/ocr2keeper/logs_integration_test.go

core/services/ocr2/plugins/ocr2keeper/evm21/logprovider/abi.go

core/services/ocr2/plugins/ocr2keeper/evm21/logprovider/basetypes_copy.go

cl-sonarqube-production · 2023-07-05T15:47:50Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

81.8% Coverage
0.0% Duplication

* log event provider (wip) * fixes 2.1 integration * log event provider + buffer * log event provider: integration test * adding a temp copy from ocr2keepers * increase timeout to stablize test TODO: figure this out w/o timeouts at all * use log struct for check data * move log event provider int test into evm21 * extract 2.1 int tests * aligned default config should fix backfill test instability * fixes and tests * lint * extract logprovider package * extract log packer * stablize test * apply lookback buffer after rate limiting * stablize tests * log struct packing

* add an example contract for mercury and log trigger * refactor * refactor * refactor * add a VL contract for log triggered feed lookup upkeep * Change update transmitter balance behaviour (#9624) * Change update transmitter balance behaviour * add test * regen go wrappers and address comments * fix tests * skip coverage for flaky test * remove todo * cleanup merge * regen wrapper * revert ine removal * skip coverage --------- Co-authored-by: FelixFan1992 <fankejin@gmail.com> * Upkeep manager refactor (#9637) * refactor upkeep administrative manager * remove only * fix * fix go build * regen interface * remove only * fix tests * addressed comments * refactor * fix tests * fix error string * fix tests * fix chaincli * Fixes When to Run Live Tests (#9666) * Update AWS action (#9672) * update VL contracts to work with 2.1 (#9673) * update VL contracts to work with 2.1 * fix * Move EVM node pool config (#9656) * Move EVM node pool config * Update after merge * Add test * Update after merge * Cleanup pipeline data references and change back to checkData (#9650) * Cleanup pipeline data references and change back to checkData * fix tests * regen wrappers * regen wrappers * Mercury feeds starts at the first block number not at zero (#9664) * Mercury feeds starts at the first block number not at zero * Update integration test * chainlink-relay => bbcb3a9 * Update core/services/relay/evm/mercury/data_source.go Co-authored-by: Sergei Drugalev <sergei.drugalev@smartcontract.com> * Fix lint * Fix test --------- Co-authored-by: Sergei Drugalev <sergei.drugalev@smartcontract.com> * [Functions] Allowlist support (#9635) * Update mercury telemetry version (#9676) * [Functions] Mark duplicate request ID log as consumed (#9647) Do it to avoid a flood of repetitive error when LogBroadcaster re-submits the same log in a loop. * Refactoring S4 plugin (#9643) * Functions/S4 integration * Fixed whitespace * Removed integration code * Reduce MAX_PERFORM and MAX_CHECK (#9657) * Properly scope logpoller to feedID for mercury (#9680) * [Functions] Use byte array for secrets in RequestData structure (#9677) Co-authored-by: Morgan Kuphal <87319522+KuphJr@users.noreply.github.com> * bump chainlink-env for pod ready improvemeents (#9640) * bump chainlink-env for pod ready improvemeents * bump one more time * function update fix * cleanup checkupkeep and callback (#9651) * cleanup checkupkeep and callback * cleanup * further cleanup * update comment * cleanup * add max revert data size struct * compile * regen wrappers and fix test * fix tests * add test for revert size limit * fix bug * add callback tests * more callback tests * more tests * regen wrappers * fix test * add clarifying comment * fix test * fix chaincli * regen wrappers --------- Co-authored-by: FelixFan1992 <fankejin@gmail.com> * Cleanup TODOs (#9682) * [BCI-1435] Improve EVM Client Readability (#9662) * added type assertion to client * moved chain and client interfaces to common * updated mocks * Added concrete interface of methods in evm client * renamed commontypes to types for headbroadcaster * removed redundent declaration of function in evm client * refactored commontypes to types for txmgr --------- Co-authored-by: Prashant Yadav <34992934+prashantkumar1982@users.noreply.github.com> * Custom configPoller to support multiple Functions plugins (#9648) * custom configPoller to support multiple Functions plugins * removed log * Fixed tests * fixed lint errors * rolled back unneeded lint changes * addressed feedback * fixed error * use imported ConfigPoller interface --------- Co-authored-by: Bolek <1416262+bolekk@users.noreply.github.com> * Use make (#9687) * fix flat fee micro link (#9690) * update url example for mercury (#9691) * Modles.go evm prefix cleanup (#9654) * Modles.go evm prefix cleanup * removed evm prefix for transmitchecker --------- Co-authored-by: Prashant Yadav <34992934+prashantkumar1982@users.noreply.github.com> * [BCF-2299] ocr2 testing (#9684) * Add integration testing OCRV2 helper functions for forwarders * Add forwarders enabled flag to ocr2 smoke test setup func * Add ocr2 with fwds smoke test * Fix forwarders models definitions to match returned json * Fix OCR2TaskJobSpec String method to include ForwardingAllowed field * Change ocr2 fwds smoke test to run for 100 rounds * Improve ocr2_helpers by removing DeployOCRv2Contracts code duplicates * Improve ocr2_helpers by removing CreateOCRv2Jobs code duplicates * Fix fwdrs ocr2 smoke test round val assertion to be of same type * add iLogAutomation geth wrapper (#9665) * Cleanup setConfig on registry (#9692) * Cleanup setConfig * regen wrappers * Allow zero address to be used as proposed admin address (#9695) * Allow zero address to be used as proposed admin address * regen wrappers * fix test * fix reorg upkeep protection to allow bypass by empty blockhash (#9699) * Fix reorg protection to allow bypass via empty blockhash * regen wrappers --------- Co-authored-by: Ryan Hall <RyanRHall@users.noreply.github.com> * Pause checkUpkeep when registry is paused (#9704) * Pause checkUpkeep when registry is paused * regen wrappers * fix test * regen wrappers * cleanup block - conditional trigger naming to be consistent (#9705) * Cleanup block and conditional trigger to be consistent * regen wrappers * regen wrappers * Bump chainlink-starknet (includes latest caigo) (#9137) * bump caigo * Use caigo to encode the point in a canonical way * Update chainlink-starknet * clean up some logging (#9675) * Automation Registrar 2.1 (#9701) * add deployRegistry21 helper * write automation registrar v2.1 * wip * set verify to false for contract verification (#9716) * Remove excess bytes in logging (not necessary to log payload) (#9711) * Functions: connection handler (#9693) * Functions: connection handler prep work * S4 methods * FunctionsConnector tests * Fixed broken test * Addressed PR feedback * Allowlist refactor * Addressed PR feedback * Preparation for Threshold plugin integration with Functions (#9670) * squashing all commits * integrated changes from #9677 * fixed lint error * fixed lint error * fixed proto name conflict * ran protoc * addressed feedback * Addressed feedback * Added decryptionQueueConfig to integration test jobspec * Fix bugs from merge conflict resolution * Initializing S4 plugin (#9725) * Initializing S4 plugin * Addressed PR feedback * Make all error keys in logs consistent (#9718) * clean up evm 20 (#9741) * add a verifiable load contract for log trigger upkeeps * fix encoding issues in registry 2.1 (#9740) * Log event provider (#9583) * log event provider (wip) * fixes 2.1 integration * log event provider + buffer * log event provider: integration test * adding a temp copy from ocr2keepers * increase timeout to stablize test TODO: figure this out w/o timeouts at all * use log struct for check data * move log event provider int test into evm21 * extract 2.1 int tests * aligned default config should fix backfill test instability * fixes and tests * lint * extract logprovider package * extract log packer * stablize test * apply lookback buffer after rate limiting * stablize tests * log struct packing * Registry 2 1 utils contract (#9727) * build AutomationUtils contract and update tests * use report struct as rawReport bytes * update wrappers * add Log struct --------- Co-authored-by: FelixFan1992 <fankejin@gmail.com> * add verifiable load contract for log trigger upkeeps * generate go wrappers * clean up * updates * format * refactor * update * remove build info * format * update * update go script to read results --------- Co-authored-by: Akshay Aggarwal <71980293+infiloop2@users.noreply.github.com> Co-authored-by: Adam Hamrick <adam.hamrick@smartcontract.com> Co-authored-by: george-dorin <120329946+george-dorin@users.noreply.github.com> Co-authored-by: Sam <samsondav@protonmail.com> Co-authored-by: Sergei Drugalev <sergei.drugalev@smartcontract.com> Co-authored-by: Bolek <1416262+bolekk@users.noreply.github.com> Co-authored-by: Andrei Smirnov <andrei.smirnov@smartcontract.com> Co-authored-by: kylesmartin <54827727+kylesmartin@users.noreply.github.com> Co-authored-by: Morgan Kuphal <87319522+KuphJr@users.noreply.github.com> Co-authored-by: Tate <tate.exon@smartcontract.com> Co-authored-by: Chia Yong Kang <chiayongkang@hotmail.com> Co-authored-by: Prashant Yadav <34992934+prashantkumar1982@users.noreply.github.com> Co-authored-by: Simson <simsonraj.easvarasakthi@smartcontract.com> Co-authored-by: ilija42 <57732589+ilija42@users.noreply.github.com> Co-authored-by: Ryan Hall <RyanRHall@users.noreply.github.com> Co-authored-by: Blaž Hrastnik <blaz@mxxn.io> Co-authored-by: Lei <lei.shi@smartcontract.com> Co-authored-by: Amir Y <83904651+amirylm@users.noreply.github.com>

amirylm force-pushed the AUTO-2565-logs-provider-for-log-trigger branch from 1bc7807 to 114e925 Compare June 14, 2023 07:12

ferglor reviewed Jun 14, 2023

View reviewed changes

reductionista reviewed Jun 14, 2023

View reviewed changes

core/services/ocr2/plugins/ocr2keeper/evm21/log_event_provider.go Outdated Show resolved Hide resolved

amirylm force-pushed the AUTO-2565-logs-provider-for-log-trigger branch from 824e292 to bdceb8c Compare June 15, 2023 13:23

reductionista reviewed Jun 15, 2023

View reviewed changes

amirylm force-pushed the AUTO-2565-logs-provider-for-log-trigger branch 5 times, most recently from 510f2a4 to 69c2636 Compare June 22, 2023 17:13

FelixFan1992 reviewed Jun 23, 2023

View reviewed changes

amirylm marked this pull request as ready for review June 23, 2023 10:44

amirylm requested a review from a team as a code owner June 23, 2023 10:44

infiloop2 reviewed Jun 23, 2023

View reviewed changes

core/services/ocr2/plugins/ocr2keeper/integration_test.go Outdated Show resolved Hide resolved

amirylm force-pushed the AUTO-2565-logs-provider-for-log-trigger branch from 69c2636 to 53e96e7 Compare June 23, 2023 10:45

infiloop2 reviewed Jun 23, 2023

View reviewed changes

core/services/ocr2/plugins/ocr2keeper/logs_integration_test.go Outdated Show resolved Hide resolved

infiloop2 previously approved these changes Jun 23, 2023

View reviewed changes

amirylm added this pull request to the merge queue Jun 23, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 23, 2023

FelixFan1992 added this pull request to the merge queue Jun 23, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 23, 2023

FelixFan1992 added this pull request to the merge queue Jun 23, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 23, 2023

FelixFan1992 added this pull request to the merge queue Jun 23, 2023

FelixFan1992 removed this pull request from the merge queue due to a manual request Jun 23, 2023

FelixFan1992 added this pull request to the merge queue Jun 23, 2023

EasterTheBunny previously approved these changes Jun 23, 2023

View reviewed changes

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 23, 2023

stablize tests

68f9526

FelixFan1992 reviewed Jul 5, 2023

View reviewed changes

core/services/ocr2/plugins/ocr2keeper/evm21/logprovider/abi.go Outdated Show resolved Hide resolved

FelixFan1992 reviewed Jul 5, 2023

View reviewed changes

core/services/ocr2/plugins/ocr2keeper/evm21/logprovider/basetypes_copy.go Show resolved Hide resolved

log struct packing

f2c3e41

amirylm enabled auto-merge July 5, 2023 16:20

amirylm requested a review from infiloop2 July 5, 2023 16:41

FelixFan1992 approved these changes Jul 5, 2023

View reviewed changes

amirylm added this pull request to the merge queue Jul 5, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 5, 2023

amirylm added this pull request to the merge queue Jul 6, 2023