How to bootstrap SpiceDB's datastore when migrating from another auth system to AuthZed?

[andyren0110]

Our application currently stores data models used for making authz decisions in couple of tables. In order to migrate to AuthZed, we need to bootstrap the relationship data from some existing data (a few million relationship tuples).

What's the best way of doing so? I see a couple of options:

1. I could have a script that goes through those few million relationship tuples and call zed cli to create the tuples.
2. I could use some data pipeline to directly write to the underlying datastore (Postgres in our case). I looked at the raw relation_tuple table. It looks like it's pretty straight forward but I'm not entirely sure how to deal with created_xid and deleted_xid columns.
3. I could use the experimental bulk api.

Or there might be some other ways.

Any suggestions are welcome!

[vroldanbet]

Hi @andyren0110 👋🏻

• ⚡ The BulkImport API is the recommended way. It leverages the bulk loading features on the underlying databases, the most efficient way to load large chunks of data. It exposes a bidirectional gRPC streaming API, so you can even parallelize data loading. We've seen this be able to ingest very quickly into Postgres instances. You can also run bulk import using zed CLI
• 👌🏻 Alternatively, you can use zed relationship create/touch command, which supports reading from STDIN, so you can have a script that generates the relationships and pipe it into zed. This comes with all the datastore's transaction ACID overhead, so it would be slower, but has the nicety that it can be implemented with good ol' shell scripts
• 🙅🏻 It's not worth the trouble of writing directly into the underlying datastore, you can get that same level of efficiency by using BulkImport which uses PG's bulk loading primitives. That tricky bit is getting SpiceDB's MVCC implementation right, which leverages some features in Postgres. As an example, not even a pg_dump/pg_load works today because the values of the transaction IDs are maintained internally by a Postgres instance. The consequence is that SpiceDB won't be able to see the tuples as it starts querying the datastore. There may be ways around it, but honestly, this is why bulk import was introduced, so that clients don't have to deal with SpiceDB internals, which may change at any time.