Create a seperate table for storing logs. #102

arijitAD · 2021-08-05T05:32:06Z

No description provided.

i-norden

Need to make some changes to the db schema, which will necessitate updating the go code some too, but otherwise everything looks great (and would work for the current schema).

core/types/receipt.go

statediff/db/migrations/00016_create_eth_log_cids_table.sql

statediff/indexer/ipfs/ipld/eth_receipt.go

statediff/indexer/ipfs/ipld/shared.go

ashwinphatak · 2021-08-11T05:59:23Z

statediff/db/migrations/00016_create_eth_log_cids_table.sql

+    topic0s               VARCHAR(66)[],
+    topic1s               VARCHAR(66)[],
+    topic2s               VARCHAR(66)[],
+    topic3s               VARCHAR(66)[],


Don't we need another column for data?

Didn't see this til after commenting: #102 (comment)

If you think it is useful from a users/consumers perspective let's do it @ashwinphatak

I think it's useful if we implement the GetLogs GQL API using the log index table in cerc-io/ipld-eth-server#92, so we don't have to read it from elsewhere?

We haven't done that yet in the linked PR in ipld-eth-server. We should return the log CID and ipld block there, maybe also the receipt CID in a new field.

@i-norden While doing this, would it make sense to filter out "failed" logs (based on the receipt status) here: https://github.com/vulcanize/ipld-eth-server/blob/master/pkg/graphql/graphql.go#L1031 (or in the table join, as long as the log index turns out correct)

i-norden

LGTM, just one or two minor tweaks and then there are a few things still open to discussion. I'll bring these things up in slack and we can figure out what the best approach is.

core/types/receipt.go

statediff/db/migrations/00016_create_eth_log_cids_table.sql

statediff/indexer/indexer.go

statediff/indexer/ipfs/ipld/eth_receipt.go

statediff/indexer/writer.go

i-norden · 2021-08-20T01:53:36Z

statediff/db/migrations/00016_create_eth_log_cids_table.sql

    log_data            BYTEA,
-    mh_key              TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,


So we still want a mh_key and cid that reference an IPLD block in this table, but instead of referencing a Log IPLD it should reference a LogTrie leaf node IPLD (the leaf node that contains the log this table was previously directly referencing). Does that make sense? It will be a bit involved/tricky to generate this cid/mh_key, you'll probably need to iterate over the trie we generate in processLogs while noting the path of each leaf node (the path is the RLP encoding of the log's index in the Receipt.Logs list, so we can tell which log is in which leaf based on the leaf node's path and the known index of the log).

i-norden

Thanks Arijit, this is looking good. A few comments, nothing major. But I'm curious if we have proper unit test coverage for the indexing of the logs table- tests that include the new mh_key and cid rows? I don't see it in this diff, might have missed it.

i-norden · 2021-08-24T02:09:16Z

statediff/indexer/indexer.go

 			}
+			logMhKey[idx] = mhKey
+			logCID[idx] = id


Do we need to reallocate the log CIDs to this new logCID slice, can't we use the slice in args.logLeafNodeCIDs?

i-norden · 2021-08-24T02:11:04Z

statediff/db/migrations/00016_create_eth_log_cids_table.sql

@@ -1,6 +1,8 @@
 -- +goose Up
 CREATE TABLE eth.log_cids (
    id                  SERIAL PRIMARY KEY,
+    cid                 TEXT NOT NULL,


I think we should rename this to leaf_cid, to make it clear our CID no longer references a log directly but rather the leaf node that contains it. What are your thoughts on that @ashwinphatak?

Yes, makes sense.

i-norden · 2021-08-24T02:11:43Z

statediff/db/migrations/00016_create_eth_log_cids_table.sql

@@ -1,6 +1,8 @@
 -- +goose Up
 CREATE TABLE eth.log_cids (
    id                  SERIAL PRIMARY KEY,
+    cid                 TEXT NOT NULL,
+    mh_key              TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,


And same here, we could rename this to leaf_mh_key to make an explicit distinction.

i-norden · 2021-08-24T02:12:58Z

statediff/indexer/indexer.go

 			mappedContracts[l.Address.String()] = true
 			logDataSet[idx] = &models.LogsModel{
 				ID:      0,
 				Address: l.Address.String(),
 				Index:   int64(l.Index),
 				Data:    l.Data,
+				CID:     logCID[idx],


Related to #102 (comment), couldn't this be args.logLeafNodeCIDs[i][idx]?

arijitAD · 2021-08-24T11:41:32Z

Will merge this PR once we have a major release #107

arijitAD marked this pull request as draft August 5, 2021 05:32

arijitAD changed the base branch from v1.10.6-statediff to v1.10.5-statediff August 5, 2021 05:32

arijitAD changed the base branch from v1.10.5-statediff to v1.10.5-statediff-0.0.25 August 5, 2021 05:33

arijitAD changed the base branch from v1.10.5-statediff-0.0.25 to v1.10.4-statediff August 5, 2021 05:33

arijitAD force-pushed the log-table branch 2 times, most recently from 4c43c21 to fdedb5f Compare August 9, 2021 11:53

arijitAD changed the base branch from v1.10.4-statediff to v1.10.6-statediff August 9, 2021 11:55

i-norden requested changes Aug 11, 2021

View reviewed changes

ashwinphatak reviewed Aug 11, 2021

View reviewed changes

arijitAD marked this pull request as ready for review August 16, 2021 12:58

arijitAD requested review from ashwinphatak and i-norden August 16, 2021 13:02

i-norden approved these changes Aug 18, 2021

View reviewed changes

i-norden self-requested a review August 20, 2021 01:47

i-norden requested changes Aug 20, 2021

View reviewed changes

i-norden mentioned this pull request Aug 20, 2021

Separate out logs into own table #79

Closed

arijitAD requested a review from i-norden August 23, 2021 16:42

i-norden reviewed Aug 24, 2021

View reviewed changes

i-norden self-requested a review August 24, 2021 02:18

i-norden approved these changes Aug 24, 2021

View reviewed changes

ashwinphatak approved these changes Aug 24, 2021

View reviewed changes

Create a seperate table for storing logs

79857f8

arijitAD force-pushed the log-table branch from ddd5490 to 79857f8 Compare August 24, 2021 12:32

arijitAD changed the base branch from v1.10.6-statediff to v1.10.8-statediff August 24, 2021 12:33

arijitAD force-pushed the log-table branch 2 times, most recently from 09e4dae to 921242a Compare August 26, 2021 16:32

Self review

ca7f7d7

arijitAD force-pushed the log-table branch from 921242a to ca7f7d7 Compare August 26, 2021 17:14

arijitAD merged commit 639af8b into v1.10.8-statediff Aug 27, 2021

github-cerc-io deleted the log-table branch October 2, 2023 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a seperate table for storing logs. #102

Create a seperate table for storing logs. #102

arijitAD commented Aug 5, 2021 •

edited

Loading

i-norden left a comment •

edited

Loading

ashwinphatak Aug 11, 2021

arijitAD Aug 16, 2021

i-norden Aug 18, 2021

ashwinphatak Aug 20, 2021

ashwinphatak Aug 20, 2021 •

edited

Loading

i-norden left a comment

i-norden Aug 20, 2021 •

edited

Loading

arijitAD Aug 23, 2021

i-norden left a comment •

edited

Loading

i-norden Aug 24, 2021

arijitAD Aug 24, 2021

i-norden Aug 24, 2021 •

edited

Loading

ashwinphatak Aug 24, 2021

arijitAD Aug 24, 2021

i-norden Aug 24, 2021 •

edited

Loading

arijitAD Aug 24, 2021

i-norden Aug 24, 2021 •

edited

Loading

arijitAD Aug 24, 2021

arijitAD commented Aug 24, 2021

		log_data BYTEA,
		mh_key TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,

Create a seperate table for storing logs. #102

Create a seperate table for storing logs. #102

Conversation

arijitAD commented Aug 5, 2021 • edited Loading

i-norden left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashwinphatak Aug 20, 2021 • edited Loading

Choose a reason for hiding this comment

i-norden left a comment

Choose a reason for hiding this comment

i-norden Aug 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden Aug 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden Aug 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden Aug 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arijitAD commented Aug 24, 2021

arijitAD commented Aug 5, 2021 •

edited

Loading

i-norden left a comment •

edited

Loading

ashwinphatak Aug 20, 2021 •

edited

Loading

i-norden Aug 20, 2021 •

edited

Loading

i-norden left a comment •

edited

Loading

i-norden Aug 24, 2021 •

edited

Loading

i-norden Aug 24, 2021 •

edited

Loading

i-norden Aug 24, 2021 •

edited

Loading