feat: update to sql engines code

[Light2Dark]

📝 Summary

I branched out to experiment. mo.sql can execute most SQL statements. Added some frontend.

Todo:

• no longer existing connections aren't being deleted
• update selection input display, icons etc (see @/mscolnick's comment below) [might not do this here]
• fix/add some tests
• add some docs on how to use this
• sql regex doesn't handle different ordering of engine & output params [probs won't do it now too]

future work:

• handle data source panel, sql parsing, so that dataset info is obtained from custom sql engines

🔍 Description of Changes

• by default, frontend will set duckdb as engine (this is hardcoded in frontend, not provided by backend)
• added a new atom to store the latestEngineSelected, so it's sort of "smart" when you create new cells

📋 Checklist

• I have read the contributor guidelines.
• For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
• I have added tests for the changes made.
• I have run the code and verified that it works as expected.

📜 Reviewers

@akshayka OR @mscolnick

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
marimo-storybook	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 30, 2025 0:48am

[vercel]

@Light2Dark is attempting to deploy a commit to the marimo Team on Vercel.

A member of the Team first needs to authorize it.

[mscolnick]

up to you on design, i was thinking we could do a non-native <Select>. we could make this fancier with logos (default to lucide database). and then put the help in the bottom (styled differently than the rest of the items).

we can also do this in a followup

[Light2Dark]

hmm, yeah thinking of doing it in a follow up. Need to hook it up to the code execution too.

could I get your thoughts on dataSourceConnections being lost on refresh? Should this be saved in local storage / somewhere

[mscolnick]

that info should be stored in the session_View.py. it looks like i started to add it, but didnt quite finish:

    @property
    def operations(self) -> list[MessageOperation]:
        all_ops: list[MessageOperation] = []
        if self.cell_ids:
            all_ops.append(self.cell_ids)
        if self.variable_operations.variables:
            all_ops.append(self.variable_operations)
        if self.variable_values:
            all_ops.append(
                VariableValues(variables=list(self.variable_values.values()))
            )
        if self.datasets.tables:
            all_ops.append(self.datasets)
        all_ops.extend(self.cell_operations.values())
        if self.stale_code:
            all_ops.append(self.stale_code)
+     # need to add append self.datasets
        return all_ops

i just quickly added this copying what i did for DataSourceConnections, which may not have been the correct API. so you may change the Datasets class if you need. (i.e. clear_channel might be useless)

[mscolnick]

could this be

code = `_df = mo.sql(f"SELECT * FROM ${table.name} LIMIT 100", engine=${table.source_type})`;

I guess none will appear in that sidebar for now? so can wait

[mscolnick]

@Light2Dark, this is awesome! i am going to merge since this is in a good state and we can keep branching off the main branch