poc: add databases in datasource

[Light2Dark]

📝 Summary

---

TODOs:

• the frontend, datasources-panel.tsx needs work, quite tricky :/
• need to filter variables and remove old db,tables
• add tests, fix typing
• stress test with large db
• test with different db's. The worst thing to do is it hangs when there's an error in the bg. It should fail and return no data

🔍 Description of Changes

Experimenting a little, I find an optimal request solution is something like below. Make an initial req for db, schemas and table list, upon expanding a table, make another request to get column info, PKs, indexes. This avoids too long of an initial request time while fetching sufficient info. With snowflake sample db (7 schemas, 10+ tables each), initial request takes about 3s. If I fetched all data, this would take ~25s

📋 Checklist

• I have read the contributor guidelines.
• For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
• I have added tests for the changes made.
• I have run the code and verified that it works as expected.

📜 Reviewers

@akshayka OR @mscolnick

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
marimo-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 15, 2025 4:43pm
marimo-storybook	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 15, 2025 4:43pm

[mscolnick]

@Light2Dark, a user on Discord had 100+ tables. We don't need to lazy load tables too just yet, but could be worth keeping a lookout / design in such a way to support it later. I do think by avoiding loading the columns its only 2 round trips (instead of a fan-out).

[mscolnick]

let me know if its worth wrapping DeferredRequestRegistry in an LRU cache. happy to write a class for that if you'd want

[mscolnick]

adding the new objects to marimo/_cli/development/commands.py should result in a cleaner yaml file

[Light2Dark]

should be kinda clean already, the objects show up nicely 🤔

[mscolnick]

oh shoot, sorry i misread this

[Light2Dark]

hey @mscolnick, the frontend seems a little tricky. Is there a reason you're putting the following state in jotai? Instead of local to a component

function initialState(): DatasetsState {
  return {
    tables: [],
    expandedTables: new Set(),
    expandedColumns: new Set(),
    columnsPreviews: new Map(),
  };
}

referring to the expanded state

[mscolnick]

@Light2Dark, jotai was originally used because request and responses came from different places. if you want to useAsyncData and keep the data local, that is fine. but we would need to handle caching too.