docs: RFC for keyspace

docs/design/2022-12-06-Keyspace.md

@@ -0,0 +1,137 @@
+# Proposal: Keyspace
+* Authors: [ystaticy](https://github.com/ystaticy), [iosmanthus](https://github.com/iosmanthus), [AmoebaProtozoa](https://github.com/AmoebaProtozoa),[zeminzhou](https://github.com/zeminzhou)
+
+
+## Motivation
+We hope to store data of multiple applications in a TiKV cluster.
+For example in multi-tenant scenario. A common solution is to add a prefix in front of the key to distinguish the data of multiple applications.
+So we introduce a new concept 'Keyspace' to describe the logical of the isolation with different business scenarios in a TiKV cluster.
+
+## Architecture
+
+This section we will describe the Architecture of the framework when using keyspaces.
+
+![keyspace-arch.png](imgs/keyspace-arch.png)
+
+### PD
+
+PD is responsible for Keyspace Meta managment. It provide http interface and RPC interface to create, load, change state or configuration keyspace meta. All other components get the keyspace meta from PD when they read and write keyspace data.
+
+### TiDB
+
+After creating the keyspace, a specified keyspace can be configured on more than one TiDB instance, which forms the TiDB cluster of the specified keyspace. However, a TiDB instance cannot serve more than one keyspace at a time. 
+
+### Client-go
+
+Client-go in TiDB uses codec V2 to add keyspace key prefix when the keyspace is set, the current TiKV client can only read and write data from the current keyspace.
+
+### TiKV
+
+In TiKV, different keyspace will distinguish by different data key prefix.
+
+### TiFlash
+
+In TiFlash, requests for TiFlash also carry keyspace prefix and data structures and storage paths in TiFlash also segregated by keyspaceID.
+
+## Usage
+
+To use TiDB with keyspace, the following configuration is required: 
+1. TiKV need to enable API V2.
+2. The TiDB configuration set `keyspace-name` to specify which keysapce is currently served by the TiDB config.
+3. BR, Lightning, Dumpling, CDC specifies the keyspace by setting keyspace-name on the command line.
+
+## Implementation
+
+### PD
+PD provide http interface and RPC interface to create, load, change state or configuration keyspace meta, and responsible for interacting with the keyspace meta store. The user requests the create keyspace interface by specifying the keyspace name, and the keyspace meta is generated. Keyspace ID is allocated by PD createKeyspace interface. The keyspace id is assigned by auto-incrementing the id, so keyspace ids are not re-assigned. The max keyspace id is 16777216, no new keyspaces can be created after the maximum keyspace id has been reached.
+
+### TiDB
+
+#### Using API V2 and etcd namespace to distinguish TiKV data and etcd path by Keyspace.
+
+1. Configure Keyspace name by setting `keyspace-name` in the configuration file.
+   * When TiDB server start, it will open a TiKV driver. The driver have a TxnKV client object `tikv.KVStore`. If `keyspace-name` has been set, it will start to use `tikv.KVStore` by API V2 to access TiKV.
+2. The etcd path should has different prefix by etcd namespace.
+   * There is a etcd client be created when domain init. If `keyspace-name` has been set, it will add a etcd namespace when create etcd client.
+   * The format of etcd namespace is `/keyspaces/tidb/$keyspaceId`.
+
+#### TiDB BR
+
+If "keyspace-name" is set when executing the BR command, it support restore and backup the specified keyspace data. BR task will parse old keyspace prefix from source data, and get target keyspace from KVStore.codec. The old keyspace prefix and new keyspace prefix is used to update old key prefix and new key prefix of rewrite rules. Restore checksum also requires the old and new keyspace prefixes. Backup task need add target keyspace prefix when build ranges in `BackupRequest` to scan out the data of the specified keyspace in TiKV.
+
+#### TiDB Lightning
+
+If "keyspace-name" is set when executing the Lightning command, it will encode key to add keyspace prefix before appends rows to the SST file, and decode key before records conflicts data into TiDB table.  
+
+#### TiDB GC
+
+Previously there was only one GCWorker for GC in the whole cluster. Each keyspace has its own GCWorker leader that can separately advance its own GC safe point, safe point information stored in PD, It is also isolated by keyspace prefix, and only cleans the data below its keyspace range, without affecting each other.
+
+### TiKV
+
+Keyspace can only be used when TiKV [enabled API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md#tikv-server).
+See more details about keyspace in TiKV in [API V2 RFC](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md).
+
+#### Keyspace Key encoding:
+
+We use [Keyspace key encoding](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md#key-encoding) to distinguish different Keyspace.When TiDB enabled Keyspace, it uses TxnKV mode prefix. 'x' is the TxnKV key mode prefixes, after mode prefix is 3 bytes for keyspace. So there are 4 bytes in key more than the scenario which does not use Keyspace. It will start with 'x' + keyspace Id (3 bytes).
+
+#### Client-go
+
+Client-go should also support API V2 and Keyspace, which will make all the codec logic transparent to upper users. We currently maintain these implementations in [tikv/client-go/api-v2](https://github.com/tikv/client-go/tree/api-v2).
+
+The new version of client-go unifies all the codec logic by introducing a new interface call `Codec`.
+```go
+// Codec is responsible for encode/decode requests.
+type Codec interface {
+	// GetAPIVersion returns the api version of the codec.
+	GetAPIVersion() kvrpcpb.APIVersion
+	// GetKeyspace return the keyspace id of the codec in bytes.
+	GetKeyspace() []byte
+	// GetKeyspaceID return the keyspace id of the codec.
+	GetKeyspaceID() KeyspaceID
+	// EncodeRequest encodes with the given Codec.
+	// NOTE: req is reused on retry. MUST encode on cloned request, other than overwrite the original.
+	EncodeRequest(req *tikvrpc.Request) (*tikvrpc.Request, error)
+	// DecodeResponse decode the resp with the given codec.
+	DecodeResponse(req *tikvrpc.Request, resp *tikvrpc.Response) (*tikvrpc.Response, error)
+	// EncodeRegionKey encode region's key.
+	EncodeRegionKey(key []byte) []byte
+	// DecodeRegionKey decode region's key
+	DecodeRegionKey(encodedKey []byte) ([]byte, error)
+	// DecodeBucketKeys decode region bucket's key
+	DecodeBucketKeys(keys [][]byte) ([][]byte, error)
+	// EncodeRegionRange encode region's start and end.
+	EncodeRegionRange(start, end []byte) ([]byte, []byte)
+	// DecodeRegionRange decode region's start and end.
+	DecodeRegionRange(encodedStart, encodedEnd []byte) ([]byte, []byte, error)
+	// EncodeRange encode a key range.
+	EncodeRange(start, end []byte) ([]byte, []byte)
+	// DecodeRange decode a key range.
+	DecodeRange(encodedStart, encodedEnd []byte) ([]byte, []byte, error)
+	// EncodeKey encode a key.
+	EncodeKey(key []byte) []byte
+	// DecodeKey decode a key.
+	DecodeKey(encoded []byte) ([]byte, error)
+}
+```
+
+This interface will encode the key-related request right before it is sent to the TiKV/TiFlash/PD server, and decode the response right after it is received. This design should neat enough to make encoding and decoding transparent to the upper users.
+
+Typically, [`codecV2`](https://github.com/tikv/client-go/blob/239ac1b2b7fc67921b00e1d51d47f3716c2c2f0c/internal/apicodec/codec_v2.go#L41) implements the `Codec` interface and all the Keyspace related codec logic.
+The encode logic is trivial, but the decode logic is a little bit complicated.
+To achive transparency, `codecV2` will map the region range into `[0, +inf)`, if the range is not overlapped with the current keyspace, it will return an error.
+For example, if the keyspace is `[x001, x002)`, then the region range `[x003, x004)` is invalid, and `[x000, x002)` is mapped to `[0, +inf)`.
+
+client-go provides a function called `NewCodecPDClientWithKeyspace` to create a PD client with Keyspace support. It try to fetch the keyspace id with the given keyspace name from PD, and create a `codecV2` with the keyspace id.
+You could get the `Codec` within the `CodecPDClient` and use it to construct a "codeced" TiKV gRPC client.
+
+#### Coprocessor
+
+TiKV Coprocessor use the range in the request to determine which range it should scan.
+After scanning, it will transform the key/value pair into the columnar format and begin to execute the DAG request.
+However, the original code could not recognize the new format key.
+So the [`RangeScanner`](https://github.com/iosmanthus/tikv/components/tidb_query_common/src/storage/scanner.rs#L20) should be modified to support the new format key by ignore the first 4 bytes if the request is using API V2.
+
+### TiFlash
+