[Experimental] Tokens dbt project

[aalan3]

This is just an example of what the transfers would look like if we were to break it up into its own sub project. It still for example uses the the top level macros folder for all our workarounds.

The benefit of doing this is that the compile times go down by quite a lot and the project is not as heavy. This also creates somewhat of a separation of concern. If we consider going down this path further we can also refactor the shared macros/resources a bit better.

The drawback is that we need to redefine models that we reference such as tokens.erc20 or evms.info as sources which is a bit annoying. We would also need to create a new CI run for each of the sub projects but they would hopefully run faster.

I had to create some sources and copy others. For the base_sources (base tables) I symlinked it, seems like it's possible to symlink files but not folders.

Imagine for example that #4698 and basically the rest of the models under tokens/ where built like this.

[0xRobin]

correct columns are: [blockchain, contract_address, decimals, symbol]

[0xRobin]

We should be very careful of the dependency direction here.
To prevent any circular dependencies I believe we should only have raw sources in a sub-project.

raw -->-- tokens (subproject) -->-- spellbook (main project)

We can't have a structure where we have both:

tokens -->-- spellbook
tokens --<-- spellbook

For this sub project this probably means:

• moving tokens.erc20 into the tokens subproject
• moving evms.info to a shared macro??? (so it can be re-used across projects..)

note: we probably need a better name for the highest level "main" dbt-project (spellbook (main project) here)

[aalan3]

This is a good point. I think at first we can't really avoid this (imagine for dex/nft trades) for example if someone wants to build labels on top of balances or transfers it would end back up in the parent repo. Eventually though we could enforce this.

This presents another challenge of how to actually map everything and enforce it.

[0xRobin]

when thinking about more sub-projects in the future we can put all the shared base sources in it's separate model folder and add it to the model-paths: in every dbt_project.yml file.

[aalan3]

Yeah I was thinking the same, we will basically have shared models and shared macros across projects.

[jeff-dude]

closing this, but it'll be around for alex to reference as needed