life of a transaction #278
Conversation
romange
force-pushed
the
TrDoc
branch
4 times, most recently
from
e07727f to
35a8dbc
Compare
| @@ -0,0 +1,261 @@ | |||
| # Life of a transaction | |||
|
|
|||
| This document describes how Dragonfly transactions provide atomicity and serializability for its multi-key and multi-command operations. | |||
There was a problem hiding this comment.
DragonflyDB transaction algorithm provide linearizability and serializability
There was a problem hiding this comment.
what is atomicity of multi-key command, isnt it the same as Serializability?
There was a problem hiding this comment.
No, the atomicity of a multi-key command is called strict serializability. See at the bottom example of serializable but not linearizable schedule.
docs/transaction.md
Outdated
|
|
||
| Linearizability (also known as atomic consistency, strong consistency, immediate consistency) describes reads and writes on a single object (stores a single value). it doesn’t involve multiple objects. It doesn’t involve “transaction”, which groups multiple objects. It treats each operation as an atom, i.e. to take effect in a single time point, rather than a timespan. | ||
|
|
||
| Note that simple, single-key operations in Dragonfly are already linearizable. Indeed, in a shared-nothing architecture each shard-thread performs operations on its keys sequentially. |
There was a problem hiding this comment.
In a shared nothing architecture each shard-thread performs operations on its keys sequentially, therefor single-key operations in DragonflyDB are already linearizable and serializability. The compexity...
Move this line to the top where you write whats in this doc
There was a problem hiding this comment.
I think that also multi key operations ensure Linearizability in the shared nothing architecture. What we handle here is just Serializability
There was a problem hiding this comment.
no, if we user1 applies "mset x a1 y b1" and user2 applies "mset x a2 y b2", user3 may read "x=a1, y=b2" (serializable) but also "x=a2, y=b1" - also serializable. Both are not linearizable because it should be either "a1 b1" or "a2 b2"
There was a problem hiding this comment.
From the definistion you added on serializable and linearizable
Linearizability (also known as atomic consistency, strong consistency, immediate consistency) describes reads and writes on a single object (stores a single value). it doesn’t involve multiple objects.
From what I understand here it has nothing to do with the multi key command
Database can executed transactions in parallel (and the operations in parallel). Serializability guarantees the result is the same with, as if the transactions were executed one by one. i.e. to behave like executed in a serial order.
From what I understand here this exactly answers the issue that we can not return "x=a2, y=b1"
There was a problem hiding this comment.
review again this comment. You describe Linearizability (for single object) and not atomicity.
Above you write
"This document describes how Dragonfly transactions provide atomicity and serializability for its multi-key and multi-command operations."
here you describe Linearizability and below you write
The complexity rises when we need to provide strict-serializability (aka serializability and linearizability) for multiple keys.
The terms are confusing.
Its better if you use one term, stick with it and describe it. f.e if you want to decribe how Dragonfly transactions provide atomicity and serializability, than define atomicity and serializability. The Linearizability definition is redundant
docs/transaction.md
Outdated
|
|
||
| ### Linearizability | ||
|
|
||
| Linearizability (also known as atomic consistency, strong consistency, immediate consistency) describes reads and writes on a single object (stores a single value). it doesn’t involve multiple objects. It doesn’t involve “transaction”, which groups multiple objects. It treats each operation as an atom, i.e. to take effect in a single time point, rather than a timespan. |
There was a problem hiding this comment.
add - Linearizability is a recency guarantee. Once a writer sets new value, the value immediately takes effect. All readers immediately see the new value
docs/transaction.md
Outdated
|
|
||
| ## Transactions high level overview | ||
| Transactions in Dragonfly are orchestrated by an abstract entity, called coordinator. | ||
| In reality, a client connection instance takes on itself the role of a coordinator: it coordinates transactions every time it executes a redis or memcached command. The algorithm behind Dragonfly transactions is based on the [VLL paper](https://www.cs.umd.edu/~abadi/papers/vldbj-vll.pdf). |
There was a problem hiding this comment.
The coordinator is handling a single command every time till it finishes right ?
I would not say executes a command because it mixes with the execute phase below, use another word.
What is to coordinate a transaction?
There was a problem hiding this comment.
a client connection instance takes on itself the role of a coordinator.The transaction to 2 phases schedule and execution, which might consist of one or more steps. The coordinator is responsible to orchestrate this 2 phases and the steps inside.
There was a problem hiding this comment.
rewritten differently.
|
|
||
| The flow consists of two different phases: *scheduling* a transaction, and *executing* it. The execution phase may consist of one or more hops, depending on the complexity of the operation we model. | ||
|
|
||
| *Note, that only the coordinator fiber is blocked. Its thread can still execute other fibers - like processing requests on other connections or handling operations for the shard it owns. This is the advantage of adopting fibers - they allow us to separate the execution context from OS threads.* |
There was a problem hiding this comment.
One thread has multiple coordinator fibers ? Or just one coordinator fiber and one data shard?
There was a problem hiding this comment.
a thread may contain many fibers. each connection is a fiber
There was a problem hiding this comment.
Ok I read again the shared nothing doc.
A thread can contain few coordinators and only one data shard
In my head a thread was only one data shard and one coordinators, so maybe we can update the diagram I created
| As shown in the snippet below, a shard thread may receive transactions in a different sequence, so a transaction with a smaller id can be added to the tx-queue after a transaction with a larger id. If the scheduling algorithm running on the data shard, can not reorder the last added transaction, it fails the scheduling request. In that case, the coordinator reverts the scheduling operation by removing the tx from the shards, and retries the whole hop again by allocating a new sequence number. In reality the fail-rate of a scheduling attempt is low and the retries are rare (subject to contention on the keys). Note, inconsistent reordering happens when two coordinators try to schedule multi-shard transactions concurrently: | ||
|
|
||
| ``` | ||
| C1: enqueue msg to Shard1 to schedule T1 |
There was a problem hiding this comment.
C1: enqueue msg to Shard1 to schedule T1
C2: enqueue msg to Shard2 to schedule T2
C2: enqueue msg to Shard1 to schedule T2
C1: enqueue msg to Shard2 to schedule T1
|
|
||
| ## Blocking commands (BLPOP) | ||
|
|
||
| Redis has a rich api with around 200 commands. Few of those commands provide blocking semantics, which allow using Redis as publisher/subscriber broker. |
There was a problem hiding this comment.
Sorry this is to much reading for me to digest in one day.. but
Is this section related to how DragonflyDB transactions framework provide serializability for its multi-key and multi-command operations?
There was a problem hiding this comment.
no, I will move it to another document in subsequent Prs. No need to read past this point
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
No description provided.