Add support for LibSQL remote

[Fryuni]

Changes

• Allows passing a URL to ASTRO_DB_REMOTE_URL pointing to a LibSQL server instance (using any supported LibSQL protocols), to a file, or to an in-memory DB. This enables Astro DB to be used with self-hosting and air-gapped deployments.
• Accepts extra LibSQL options as query parameters in the remote URL.
  • For example, memory:?syncUrl=libsql%3A%2F%2Fdb-server.example.com would create an in-memory embedded replica for the LibSQL DB on libsql://db-server.example.com.

Testing

Replicated the test for Studio remote using the LibSQL remote

Docs

• Add guidance on how to use LibSQL with Astro DB docs#8955

[changeset-bot]

🦋 Changeset detected

Latest commit: 9d0d588

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Adammatthiesen]

Amazing how simple the changes that are required to unlock the potential of this library, I could see many people very happy about this change! since it will also allow more "Self-hosted-production" database options!

[syhily]

I'm very happy and confident that this PR unlocks new use cases for Astro DB. I hope it gets merged soon.

[ematipico]

Thank you @Fryuni. I'll make sure that Core is aware of this PR :)

[matthewp]

How is this working? Is this because the libsql db has an https URL? Any links you can share? Seems exciting.

[Fryuni]

@matthewp

How is this working?

It uses the DB itself to manage the snapshots, like Studio did back in the closed beta.

Is this because the libsql db has an https URL?

LibSQL has two APIs, one exclusively over HTTP and one using the Hrana protocol, which can work both over HTTP or over WebSockets or a combination of both.

The Hrana protocol allows for stateful connections, so transactions are supported.

Any links you can share? Seems exciting.

Besides the specs above, the supported URL protocols are described in LibSQL's type docs. It links to a section of the README that was removed during some reorganization of the repo a few months ago.

The LibSQL client uses the Hrana protocol. The given URL can have http, https, ws or wss as the protocol to select which underlying transport should be used or can be libsql and let the library decide which transport to use (currently selects HTTP). It can also be file to use a local DB as we do for dev or the exact string :memory: to use a DB without a backing file for persistence (nice for testing and as replica for a remote DB).

Since Studio already uses http/https I added the variation of libsql+http and libsql+https to select a LibSQL remote using an explicit HTTP transport.

[Fryuni]

To test it out, I have this route deployed using this branch and configured to use an embedded in-memory replica synchronizing with Turso.

Working perfectly :)

[matthewp]

@Fryuni Fred is out this week, he has the most context for how migrations work, so when he gets back he'll review this and leave any feedback.

[matthewp]

@Fryuni Plan is still to have this in for 4.13

[sarah11918]

Just noting that if this goes through, we will want docs for how someone can do this in the DB guide!

[matthewp]

!preview db-remote

[sarah11918]

Hey @Fryuni ! Friendly ping to remind you that I'd normally want to have a docs PR reviewed before approving an astro PR, so I know how to make helpful suggestions to the changeset here! And, this content is what helps Erika write the release blog post so she knows how to hype your feature!

Since PRs will be merged tomorrow for the minor, can I help you in any way to get a docs PR together today?

[Fryuni]

I got caught up with some things recently, but I'll have a PR for docs today 😁

[sarah11918]

Approving for docs! 🥳

[ematipico]

Can you expand the current comment and explain why this error doesn't need to be handled? Still, I think it's valuable to have a log of the error.

[matthewp]

It does seem a bit odd to me to reuse the environment variables for Studio to connect to something non-studio. Can we not have different variables for this use-case? Would that be a problem?

Think a little more work needed here

[Princesseuh]

We'll be releasing this outside of the minor, as there's still some concerns to be addressed.

[Fryuni]

It does seem a bit odd to me to reuse the environment variables for Studio to connect to something non-studio. Can we not have different variables for this use-case? Would that be a problem?

I explained that on Discord to Sarah, but it might have gotten lost there:

Since the Studio side is closed source, I don't know what would break internally if I changed the name of the env var. If it were just DB, I'd have renamed it to something like ASTRO_DB_REMOTE_URL and ASTRO_DB_TOKEN.
So I kept the same name to not break anything in Studio.

---

I'll refactor the code to accept different values, here is what I'm thinking:

• Add a new ASTRO_DB_REMOTE_URL, which would be used just for LibSQL support.
  • If only ASTRO_STUDIO_REMOTE_URL is present, use that URL to connect as a Studio backend.
  • If only ASTRO_DB_REMOTE_URL is present, use that URL to connect to LibSQL. The scheme would no longer need to be modified.
  • If neither env is present, use the prod Studio URL.
  • If both env vars are present, use the Studio one.
• When connecting to Studio, the token is read from ASTRO_STUDIO_TOKEN
• When connecting to LibSQL, the token is read from ASTRO_DB_TOKEN

This way, it should not change anything about configuring the Studio connection while also adding LibSQL support.

What do you guys think?

[ematipico]

@Fryuni I like the proposal! :)

[matthewp]

@Fryuni sorry for the late reply, yes that sounds good! Basically just treat this as a new feature that's unrelated to the Astro Studio flow. So your proposal sounds good.

[sarah11918]

Still approving for docs, and wondering whether this will be included in 4.15!