Docs: Filecoin data

[ccerv1]

What is the improvement or update you wish to see?

We should highlight this new dataset on our Data Portal docs.

This requires two steps:

• Create a dbt source model that includes each of the relevant tables. See here for an example. (You can keep the database as opensource-observer)

• Make a PR to our docs (here is the relevant page). You should also include a logo for Filecoin or Lily in the PR.

Is there any context that might help us understand?

Filecoin data on BigQuery now live!
Docs here: https://lilium.sh/data/

select * from `lily-data.lily.block_headers` where height > 4000000 limit 10

Does the docs page already exist? Please link to it.

https://docs.opensource.observer/docs/integrate/overview/

[ryscheng]

Awesome! We should also link to it in our data overview page.

Quick question --- is this dataset in the right region (US multi-region)?
If not, we may need to replicate it, like we did for Farcaster data. I can show you how we did that.

[davidgasquez]

Dataset is in us-east4.

[ryscheng]

Ah, in that case we'll probably need to replicate it to the US multi-region.
Would you be open to moving it there? Our pipelines require that last I checked.

If we need to replicate, here's a simple tutorial that'll make it happen:
https://docs.opensource.observer/docs/contribute/connect-data/bigquery/replication

It's similar to how we did it for Lens data
https://github.com/opensource-observer/oso/blob/main/warehouse/oso_dagster/assets/lens.py

[davidgasquez]

Yup! I can take care of that. will probably tackle it early next week.

[davidgasquez]

With #2033 merged, would the next step be creating the dbt models that read from opensource-observer.filecoin-lily?

[ryscheng]

Yes!!! I think the next step is to create the dbt sources, with dbt docs so that it shows up in our reference documentation:
models.opensource.observer.

I am currently working on fixing a bug in our Dagster credentialing that is preventing Dagster from actually creating the copy job. Auth issue, but I'll work on that in parallel

[ryscheng]

Debugging credential issues here
#2073

[ryscheng]

Published to our data exchange here
https://console.cloud.google.com/bigquery/analytics-hub/exchanges/projects/87806073973/locations/us/dataExchanges/open_source_observer_190181416ae/listings/filecoin_191c7f4bd01

[ryscheng]

Adding to our docs here
#2088

[ryscheng]

I think the last part is the dbt source file, which will help include it in our reference documentation.

@davidgasquez lmk if you need help here, excited to close this out :-)