Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Token transfers for ethereum #4633

Merged
merged 40 commits into from
Nov 7, 2023
Merged

Token transfers for ethereum #4633

merged 40 commits into from
Nov 7, 2023

Conversation

aalan3
Copy link
Contributor

@aalan3 aalan3 commented Oct 19, 2023

Related to #4521

This PR implements a transfers_base macro which materializes ERC20 transfers and native transfers into a single table and a transfers_enrichment macro. The design is based on the fungible transfers.

They are meant to be used together where the base is for materialization and the enrichment is a view on top of it. I'm not yet sure the enrichment step will work as a view performance-wise. I think prices_usd_forward_fill is a bit inefficient but we'll see.

Copy link
Collaborator

@0xRobin 0xRobin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great! Some initial comments:

models/tokens/ethereum/tokens_ethereum_transfers.sql Outdated Show resolved Hide resolved
models/tokens/ethereum/tokens_ethereum_transfers_base.sql Outdated Show resolved Hide resolved
models/tokens/ethereum/tokens_ethereum_transfers_base.sql Outdated Show resolved Hide resolved
macros/models/_sector/tokens/transfers_enrich.sql Outdated Show resolved Hide resolved
@belen-pruvost belen-pruvost force-pushed the token_transfers branch 2 times, most recently from 15c8fe1 to 511ea96 Compare October 24, 2023 07:52
@aalan3 aalan3 requested a review from 0xRobin October 27, 2023 10:41
{% if is_incremental() %}
AND {{incremental_predicate('block_time')}}
{% endif %}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you planning to add gas fees further in spells? It's also sort of transfer which affects balances

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any more insight on this? gas fees makes balances a bit tricky but aren't a lot of the fees in the traces?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can do just:

select "from" as spender, gas_price * gas_used as amount from {blockchain}.transactions

but it's better to doublecheck if this is applicable for L2 as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I will look into this in a follow up PR.

@grkhr
Copy link
Contributor

grkhr commented Nov 7, 2023

@aalan3
Copy link
Contributor Author

aalan3 commented Nov 7, 2023

@aalan3 genesis balances should be included as well!

https://github.com/duneanalytics/spellbook/blob/main/models/balances/ethereum/genesis/genesis_balances.sql

Good point. We have this issue for optimism as well . The question is, should we consider these transfers or just include them in the balances later? We would have to put dummy values for block_number, trace call etc.

@grkhr
Copy link
Contributor

grkhr commented Nov 7, 2023

@aalan3 genesis balances should be included as well!
https://github.com/duneanalytics/spellbook/blob/main/models/balances/ethereum/genesis/genesis_balances.sql

Good point. We have this issue for optimism as well . The question is, should we consider these transfers or just include them in the balances later? We would have to put dummy values for block_number, trace call etc.

I'd say both options are ok. Let's see what others think?

@aalan3 aalan3 requested review from jeff-dude and grkhr and removed request for grkhr November 7, 2023 15:39
Comment on lines +91 to +92
-- We have to create this because evt_index and trace_address can be null
, {{dbt_utils.generate_surrogate_key(['t.block_number', 'tx.index', 't.evt_index', "array_join(trace_address, ',')"])}} as unique_key
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we prefer this approach of adding surrogate key, rather than coalesce() each unique key to a default value?

this could be a decision that has an impact across spellbook. of course, we don't have to standardize it all right away, but will be nice to be consistent moving forward at minimum (and on larger sector spells)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH I don't like having unique_key in most cases but here I think it works because not sure what we should coalesce to. I think it can be confusing for users to have a value in evt_index/trace_address that doesn't lead anywhere.

I think if we do have a unique key column we should always try to use generate_surrogate_key because 1) it hashes the output so that it takes less space and 2) it works the same everywhere.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm okay with the rationale behind the first part. i will look to consider the approach in new dex.trades design, if applicable.

agree fully with the second part.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still believe we should adapt the incremental logic to use IS NOT DISTINCT FROM as equality operation so that the merge works as we would want with respect to null values.

Copy link
Member

@jeff-dude jeff-dude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm comfortable with this current state. thank you for taking in all the input from the community and helping get this moving 🚀

for those following along, keep in mind that this is in somewhat of a "beta" state and we welcome all feedback once live on how to improve. in the meantime, other blockchain versions will look to follow suit of ethereum here.

@jeff-dude jeff-dude removed the in review Assignee is currently reviewing the PR label Nov 7, 2023
@aalan3 aalan3 merged commit e930015 into main Nov 7, 2023
@aalan3 aalan3 deleted the token_transfers branch November 7, 2023 16:38
@github-actions github-actions bot locked and limited conversation to collaborators Nov 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
dune team created by dune team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants