Fix `getLog` API to use `log_cids` table #92

arijitAD · 2021-08-16T05:41:55Z

Remove ipld-eth-indexer use statediff geth instead.
Fix getLog API to use log_cids table
Fixes: Update getLogs to work with new log_cids table #87 and Lift code used from here into statediffing geth and ipld-eth-server vulcanize/ipld-eth-indexer#84

arijitAD · 2021-08-31T12:46:29Z

@ashwinphatak Can you specify the CLI flags we use to run to the server? Wanted to cross-check if the CLI is not broken by my changes.

ashwinphatak · 2021-08-31T12:49:22Z

@ashwinphatak Can you specify the CLI flags we use to run to the server? Wanted to cross-check if the CLI is not broken by my changes.

I'm using:

./ipld-eth-server serve --config=./environments/config.toml --eth-server-graphql

ashwinphatak · 2021-08-31T13:33:37Z

pkg/graphql/graphql.go

+			log:        l.Log,
+			cid:        l.CID,
+			receiptCID: l.RctCID,
+			ipldBlock:  l.RctData,


ipldBlock should be the log, not the receipt.

ashwinphatak · 2021-08-31T13:34:26Z

pkg/graphql/schema.go

@@ -72,6 +72,9 @@ const schema string = `

        # IPLD block data for the Receipt this Log exists in.
        ipldBlock: Bytes!
+
+		# Status of the Receipt IPLD block this Log exists in.


Nit: Indentation is different from above.

i-norden

LGTM, added some minor comments.

i-norden · 2021-08-31T12:53:10Z

db/migrations/00016_create_eth_log_cids_table.sql

+    leaf_cid            TEXT NOT NULL,
+    leaf_mh_key         TEXT NOT NULL REFERENCES public.blocks (key) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+    receipt_id          INTEGER NOT NULL REFERENCES eth.receipt_cids (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED,
+    address             VARCHAR(66) NOT NULL,


Just a heads up, this is not NOT NULL in the migration that is in go-ethereum

I have added a note in this task cerc-io/go-ethereum#73
I will update the schema once we have a single repo for the DB schema.

i-norden · 2021-08-31T13:00:11Z

pkg/eth/api.go

+		extractMiner = true
+	}
+
+	fields := RPCMarshalHeader(header, extractMiner)


We can just pass RPCMarshalHeader(header, pea.B.Config.ChainConfig.Clique != nil) instead of allocating this bool (here and on line 1092 and 1113), super insignificant optimization but I actually think it makes the code cleaner/less cluttered. Up to you.

i-norden · 2021-08-31T15:01:40Z

pkg/eth/ipld_fetcher.go

@@ -167,8 +167,95 @@ func (f *IPLDFetcher) FetchRcts(tx *sqlx.Tx, cids []eth.ReceiptModel) ([]ipfs.Bl
 	return rctIPLDs, nil
 }

+// FetchLogs fetches logs.
+func (f *IPLDFetcher) FetchLogs(logCIDs []logResult) ([]*types.Log, error) {


This isn't fetching anything, it's decomposing the already fetched logResults.

Updated the function name.

i-norden · 2021-08-31T15:03:02Z

pkg/eth/ipld_fetcher.go

+}
+
+// FetchGQLLogs fetches logs for graphql.
+func (f *IPLDFetcher) FetchGQLLogs(logCIDs []logResult) ([]logsCID, error) {


Same here, not actually fetching IPLDs

Updated the function name.

ashwinphatak · 2021-09-02T05:10:06Z

pkg/graphql/schema.go

@@ -67,11 +67,17 @@ const schema string = `
        # Transaction is the transaction that generated this log entry.
        transaction: Transaction

-        # CID for the Receipt IPLD block this Log exists in.
+        # CID for the Receipt IPLD block of leaf node.


Receipt => Log in the comment?

ashwinphatak · 2021-09-02T05:10:44Z

pkg/graphql/schema.go

        ipldBlock: Bytes!
+
+        # Status of the Receipt IPLD block this Log exists in.
+        status: Int!


What will this return for pre-byzantium fork records?

https://ethereum.stackexchange.com/questions/28889/what-is-the-exact-meaning-of-a-transactions-new-receipt-status-field

// Assign receipt status or post state. if len(receipt.PostState) > 0 { fields["root"] = hexutil.Bytes(receipt.PostState) } else { fields["status"] = hexutil.Uint(receipt.Status) }

@ashwinphatak
As per my understanding pre-byzantium fork, we used to return PostState which is the root and if it is not present then we return the status.
For GQL we need to change the type of status to Bytes! so it can accommodate both. Or should I add two separate fields root and status

PostState which is the root

Root of what? Is there some notion of success/failure by inspecting the PostState?

Intermediate state root when the transaction is being applied if the transaction fails
Source: ethereum/EIPs#98 (comment)
But then they removed the root field and add status in EIP-658 this field was replaced by status

As per https://eips.ethereum.org/EIPS/eip-658, "With the introduction of the REVERT opcode in EIP140, it is no longer possible for users to assume that a transaction failed iff it consumed all gas".

So, we may be able to return 1 or 0 based on whether the tx consumed all gas for pre-byzantium blocks?

@i-norden what do you think?

I don't think we can assume whether or not a tx failed based on how much of the gas limit was used.

If a tx is reverted early on in its execution, due to some programmatic failure of an OPCODE execution, then it can be reverted early on and the amount of gas used can be much lower than the limit allocated. But, if a tx reverts because there is not enough gas allocated, then it will only revert after the miner has executed all of the OPCODEs it can afford to before bumping up against the limit and realizing there is not enough left to finish execution. In this case the reverted tx may use all or most of the gas limit.

Another edge cases that make this approach infeasible is, for example, if someone vastly overestimated the gas limit needed for a tx. The tx is properly executed but done so far below the allocated limit, so using this heuristic the tx would look like it failed when it didn't.

Oh I misread, you meant for pre-byzantium blocks. So I just stated the obvious.

Theoretically it is possible that a tx uses up the entire gas limit during normal, successful execution- no? But the statement in that EIP makes it sound like it is a safe assumption.

ashwinphatak · 2021-09-03T10:40:48Z

pkg/graphql/schema.go

@@ -67,11 +67,17 @@ const schema string = `
        # Transaction is the transaction that generated this log entry.
        transaction: Transaction

-        # CID for the Receipt IPLD block this Log exists in.
+        # CID for the leaf node IPLD block of the log.
        cid: String!


This is always returning an empty string.

i-norden

Left some comments, I need some clarification on some things and some changes need to be done. Also, my main hangup is that we are breaking compatibility with the regular eth json rpc endpoint with these changes, so I think that requires some further discussion.

i-norden · 2021-09-14T21:10:08Z

pkg/eth/api.go

@@ -524,12 +524,18 @@ func (pea *PublicEthAPI) localGetTransactionReceipt(ctx context.Context, hash co
 		"logsBloom":         receipt.Bloom,
 	}

-	// Assign receipt status or post state.
-	if len(receipt.PostState) > 0 {
+	if blockNumber <= pea.B.Config.ChainConfig.ByzantiumBlock.Uint64() {


A problem with this is that this deviates from the normal, expected behavior of this eth json-rpc endpoint. The regular endpoint only fills in the status if it is explicitly known: https://github.com/ethereum/go-ethereum/blob/master/internal/ethapi/api.go#L1650. So this breaks compatibility with the regular endpoint.

i-norden · 2021-09-14T21:10:50Z

pkg/eth/api.go

+		} else {
+			fields["status"] = hexutil.Uint(types.ReceiptStatusSuccessful)
+		}
+	} else if len(receipt.PostState) > 0 {


We still want to fill in the fields["root"] if receipt.PostState is present, whether or not we can also assume the status above.

i-norden · 2021-09-14T21:12:08Z

pkg/eth/api.go

-	// Assign receipt status or post state.
-	if len(receipt.PostState) > 0 {
+	if blockNumber <= pea.B.Config.ChainConfig.ByzantiumBlock.Uint64() {
+		if receipt.GasUsed > tx.Gas() {


I'm missing something. When will receipt.GasUsed ever be more than tx.Gas()? When a tx fails for whatever reason is it's receipt.GasUsed simply set to something above the tx.Gas() to indicate it failed? Seems odd but I'm not super familiar with this pre-byzantium behavior- wouldn't it need to track/report how much gas was actually used during the failed execution/reversion (because the tx sender still has to pay for whatever execution was performed prior to it reverting/failing)?

Added an issue to track this #102

i-norden · 2021-09-14T21:15:09Z

db/migrations/00006_create_eth_transaction_cids_table.sql

@@ -10,6 +10,7 @@ CREATE TABLE eth.transaction_cids (
  src                   VARCHAR(66) NOT NULL,
  tx_data               BYTEA,
  tx_type               BYTEA,
+  gas                   INTEGER NOT NULL,


Let's call this gas_limit, it's more accurate to how things are defined in EIPs/specs and more explicit about what it is/how it is used.

This reverts commit 04a0f9a.

ashwinphatak

Ran watcher-ts smoke tests against this, tests passed.

i-norden

LGTM. We might want to go ahead and update the schema.sql, or not since we want to move to using ipld-eth-db for migrations/schema. Depends how soon you think we will remove the migrations/schema from here and rely on ipld-eth-dn. If soon then don't worry about it, if it will be some time (weeks) then go ahead and update the schema file here.

arijitAD · 2021-09-16T12:12:53Z

LGTM. We might want to go ahead and update the schema.sql, or not since we want to move to using ipld-eth-db for migrations/schema. Depends how soon you think we will remove the migrations/schema from here and rely on ipld-eth-dn. If soon then don't worry about it, if it will be some time (weeks) then go ahead and update the schema file here.

I am picking up the task for moving the schema to ipld-eth-db next. Should be done by next week.

arijitAD added 4 commits August 12, 2021 11:56

Remove ipld-eth-indexer dependency.

d8a5358

Fix get logs API.

b3e4fbf

Refactoring.

c0a4600

Update go mod.

3d1b308

arijitAD force-pushed the fix-get-logs branch from 08d7b95 to 3d1b308 Compare August 17, 2021 05:30

arijitAD changed the base branch from remove-indexer-dependecy to master August 17, 2021 05:31

arijitAD mentioned this pull request Aug 17, 2021

Remove ipld-eth-indexer dependency. #88

Closed

ramilexe mentioned this pull request Aug 18, 2021

Use ipld-eth-server RPC endpoint for eth_call vulcanize/uniswap-watcher-ts#169

Closed

ashwinphatak mentioned this pull request Aug 20, 2021

Create a seperate table for storing logs. cerc-io/go-ethereum#102

Merged

arijitAD added 3 commits August 30, 2021 21:25

Fix get log API to use log_cids table.

a28892f

Add reciept status in log for graphql.

d0c3241

Self review.

71837c4

arijitAD force-pushed the fix-get-logs branch from 16a30c7 to 71837c4 Compare August 31, 2021 12:40

arijitAD marked this pull request as ready for review August 31, 2021 12:41

arijitAD requested review from i-norden and ashwinphatak August 31, 2021 12:41

ashwinphatak reviewed Aug 31, 2021

View reviewed changes

i-norden approved these changes Aug 31, 2021

View reviewed changes

Add test for graphql.

967c148

arijitAD requested review from i-norden and ashwinphatak September 1, 2021 12:58

ashwinphatak reviewed Sep 2, 2021

View reviewed changes

arijitAD added 2 commits September 2, 2021 19:12

Change getLogs GQL API to take nil contract address.

121c75c

Minor change.

4d9edd2

ashwinphatak reviewed Sep 3, 2021

View reviewed changes

i-norden mentioned this pull request Sep 3, 2021

Use groupcache pool for state db access #91

Merged

arijitAD force-pushed the fix-get-logs branch from 0491cab to 70ccc51 Compare September 8, 2021 06:43

arijitAD force-pushed the fix-get-logs branch from 70ccc51 to 65411e7 Compare September 8, 2021 06:55

Address comments.

458aae1

arijitAD force-pushed the fix-get-logs branch from 65411e7 to 458aae1 Compare September 8, 2021 06:55

i-norden mentioned this pull request Sep 10, 2021

Btree indexes #100

Closed

arijitAD added 2 commits September 14, 2021 17:25

Add status field for pre-byzantium blocks.

04a0f9a

Add index and fix go.mod.

7f0c8fb

arijitAD requested a review from ashwinphatak September 14, 2021 12:19

i-norden requested changes Sep 14, 2021

View reviewed changes

Revert "Add status field for pre-byzantium blocks."

ed4171a

This reverts commit 04a0f9a.

arijitAD mentioned this pull request Sep 15, 2021

Explore computation of receipt status of pre byzantium blocks. #102

Closed

arijitAD requested a review from i-norden September 15, 2021 12:01

ashwinphatak approved these changes Sep 15, 2021

View reviewed changes

i-norden approved these changes Sep 16, 2021

View reviewed changes

arijitAD merged commit cf45439 into master Sep 16, 2021

arijitAD deleted the fix-get-logs branch September 16, 2021 12:13

arijitAD mentioned this pull request Sep 17, 2021

Lift code used from here into statediffing geth and ipld-eth-server vulcanize/ipld-eth-indexer#84

Closed

i-norden mentioned this pull request Sep 17, 2021

Add log index in the log table. #70

Closed

Fix getLog API to use log_cids table #92

Fix getLog API to use log_cids table #92

Conversation

arijitAD commented Aug 16, 2021 • edited Loading

arijitAD commented Aug 31, 2021

ashwinphatak commented Aug 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arijitAD Sep 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashwinphatak Sep 8, 2021 • edited Loading

Choose a reason for hiding this comment

i-norden Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

i-norden Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

i-norden Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden Sep 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

i-norden Sep 14, 2021 • edited Loading

Choose a reason for hiding this comment

ashwinphatak left a comment

Choose a reason for hiding this comment

i-norden left a comment

Choose a reason for hiding this comment

arijitAD commented Sep 16, 2021

Fix `getLog` API to use `log_cids` table #92

Fix `getLog` API to use `log_cids` table #92

arijitAD commented Aug 16, 2021 •

edited

Loading

arijitAD Sep 1, 2021 •

edited

Loading

ashwinphatak Sep 8, 2021 •

edited

Loading

i-norden Sep 13, 2021 •

edited

Loading

i-norden Sep 13, 2021 •

edited

Loading

i-norden Sep 13, 2021 •

edited

Loading

i-norden Sep 14, 2021 •

edited

Loading

i-norden Sep 14, 2021 •

edited

Loading