-
Notifications
You must be signed in to change notification settings - Fork 75
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add mirror document Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * fix svg to png Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Update docs/api/mirror-gateway.md Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * Update docs/api/mirror-gateway.md Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * Update docs/api/mirror-gateway.md Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt * Update docs/overview/component/mirror-gateway.md Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt * Apply suggestions from code review * Update docs/overview/component/mirror-gateway.md * fix expand bug for mirror tutorial Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * style: Format code with prettier and gofumpt * fix filename typo Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * add new link Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * add cluster role settings for mirror gateway Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * add hdf5 installation section Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Update docs/user-guides/mirroring-configuration.md Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> * update link Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * delete quote Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * update mirror component overview image Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * update mirror component overview image Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> * Update docs/overview/component/mirror-gateway.md Co-authored-by: Kevin Diu <kevindiujp@gmail.com> * Update docs/overview/component/mirror-gateway.md * Update docs/api/mirror-gateway.md Co-authored-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com> * Update docs/api/mirror-gateway.md * Update docs/user-guides/mirroring-configuration.md Co-authored-by: Kevin Diu <kevindiujp@gmail.com> --------- Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: Kiichiro YUKAWA <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Kevin Diu <kevindiujp@gmail.com> Co-authored-by: Yusuke Kadowaki <yusuke.kadowaki.1231@gmail.com>
- Loading branch information
5 people
committed
Sep 21, 2023
1 parent
1f24460
commit 8bbe931
Showing
9 changed files
with
730 additions
and
0 deletions.
There are no files selected for viewing
Binary file added
BIN
+26.2 KB
assets/docs/overview/component/mirror-gateway/full-mesh-connection.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| # Vald Mirror Gateway APIs | ||
|
|
||
| ## Overview | ||
|
|
||
| Mirror Service is responsible for providing the `Register` and `Advertise` interface for the Vald Mirror Gateway. | ||
|
|
||
| ```rpc | ||
| service Mirror { | ||
| rpc Register(payload.v1.Mirror.Targets) returns (payload.v1.Mirror.Targets) {} | ||
|
|
||
| rpc Advertise(payload.v1.Mirror.Targets) returns (payload.v1.Mirror.Targets) {} | ||
| } | ||
| ``` | ||
|
|
||
| ## Register RPC | ||
|
|
||
| Register RPC is the method to register other Vald Mirror Gateway targets. | ||
|
|
||
| ### Input | ||
|
|
||
| - the scheme of `payload.v1.Mirror.Targets`. | ||
|
|
||
| ```rpc | ||
| message Mirror { | ||
| message Target { | ||
| string host = 1; | ||
| uint32 port = 2; | ||
| } | ||
|
|
||
| message Targets { | ||
| repeated Target targets = 1; | ||
| } | ||
| } | ||
| ``` | ||
| - Mirror.Targets | ||
| | field | type | label | required | desc. | | ||
| | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | | ||
| | targets | Mirror.Target | repeated(Array[Mirror.Target]) | \* | The multiple target information. | | ||
| - Mirror.Target | ||
| | field | type | label | required | desc. | | ||
| | :---: | :----- | :---- | :------: | :------------------- | | ||
| | host | string | | \* | The target hostname. | | ||
| | port | uint32 | | \* | The target port. | | ||
| ### Output | ||
| - the scheme of `payload.v1.Mirror.Targets`. | ||
| ```rpc | ||
| message Mirror { | ||
| message Target { | ||
| string host = 1; | ||
| uint32 port = 2; | ||
| } | ||
| message Targets { | ||
| repeated Target targets = 1; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| - Mirror.Targets | ||
|
|
||
| | field | type | label | required | desc. | | ||
| | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | | ||
| | targets | Mirror.Target | repeated(Array[Mirror.Target]) | | The multiple target information. | | ||
|
|
||
| - Mirror.Target | ||
|
|
||
| | field | type | label | required | desc. | | ||
| | :---: | :----- | :---- | :------: | :------------------- | | ||
| | host | string | | \* | The target hostname. | | ||
| | port | uint32 | | \* | The target port. | | ||
|
|
||
| ### Status Code | ||
|
|
||
| | code | desc. | | ||
| | :--: | :---------------- | | ||
| | 0 | OK | | ||
| | 1 | CANCELLED | | ||
| | 3 | INVALID_ARGUMENT | | ||
| | 4 | DEADLINE_EXCEEDED | | ||
| | 13 | INTERNAL | | ||
|
|
||
| ## Advertise RPC | ||
|
|
||
| Advertise RPC is the method to advertise Vald Mirror Gateway targets. | ||
|
|
||
| ### Input | ||
|
|
||
| - the scheme of `payload.v1.Mirror.Targets`. | ||
|
|
||
| ```rpc | ||
| message Mirror { | ||
| message Target { | ||
| string host = 1; | ||
| uint32 port = 2; | ||
| } | ||
|
|
||
| message Targets { | ||
| repeated Target targets = 1; | ||
| } | ||
| } | ||
| ``` | ||
| - Mirror.Targets | ||
| | field | type | label | required | desc. | | ||
| | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | | ||
| | targets | Mirror.Target | repeated(Array[Mirror.Target]) | \* | The multiple target information. | | ||
| - Mirror.Target | ||
| | field | type | label | required | desc. | | ||
| | :---: | :----- | :---- | :------: | :------------------- | | ||
| | host | string | | \* | The target hostname. | | ||
| | port | uint32 | | \* | The target port. | | ||
| ### Output | ||
| - the scheme of `payload.v1.Mirror.Targets`. | ||
| ```rpc | ||
| message Mirror { | ||
| message Target { | ||
| string host = 1; | ||
| uint32 port = 2; | ||
| } | ||
| message Targets { | ||
| repeated Target targets = 1; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| - Mirror.Targets | ||
|
|
||
| | field | type | label | required | desc. | | ||
| | :-----: | :------------ | :----------------------------- | :------: | :------------------------------- | | ||
| | targets | Mirror.Target | repeated(Array[Mirror.Target]) | | The multiple target information. | | ||
|
|
||
| - Mirror.Target | ||
|
|
||
| | field | type | label | required | desc. | | ||
| | :---: | :----- | :---- | :------: | :------------------- | | ||
| | host | string | | \* | The target hostname. | | ||
| | port | uint32 | | \* | The target port. | | ||
|
|
||
| ### Status Code | ||
|
|
||
| | code | desc. | | ||
| | :--: | :---------------- | | ||
| | 0 | OK | | ||
| | 1 | CANCELLED | | ||
| | 3 | INVALID_ARGUMENT | | ||
| | 4 | DEADLINE_EXCEEDED | | ||
| | 13 | INTERNAL | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Vald Mirror Gateway | ||
|
|
||
| Vald Mirror Gateway is an optional component of the Vald, allowing the vector data to be synchronized across multiple Vald clusters. | ||
|
|
||
| This component makes it possible to enhance availability during a cluster failure. | ||
|
|
||
| <img src="../../../assets/docs/overview/component/mirror-gateway/mirror-gateway.png"> | ||
|
|
||
| ## Responsibility | ||
|
|
||
| Vald Mirror Gateway is responsible for the followings: | ||
|
|
||
| - Forward user requests ([Insert](https://vald.vdaas.org/docs/api/insert/) / [Upsert](https://vald.vdaas.org/docs/api/upsert/) / [Update](https://vald.vdaas.org/docs/api/update/) / [Remove](https://vald.vdaas.org/docs/api/remove/)) to the other Vald Mirror Gateways in the same group. | ||
| - Manages the state of indexes stored in all clusters to ensure they are consistent. | ||
|
|
||
| ## Features | ||
|
|
||
| This chapter shows the main features to fulfill Vald Mirror Gateway’s role: | ||
|
|
||
| - Full mesh connection | ||
| - Request forwarding | ||
| - Automatic rollback on failure | ||
|
|
||
| ### Full mesh connection | ||
|
|
||
| <img src="../../../assets/docs/overview/component/mirror-gateway/full-mesh-connection.png"> | ||
|
|
||
| The Vald Mirror Gateway is designed to interconnect with Vald Mirror Gateways in other Vald clusters. | ||
|
|
||
| Vald Mirror Gateway uses a Custom Resource called the `ValdMirrorTarget` to manage the connection destination information between Vald Mirror Gateways. | ||
|
|
||
| The `ValdMirrorTarget` is a Custom Resource related to the connection destination to other Vald Mirror Gateway. | ||
|
|
||
| When two Vald clusters contain Vald Mirror Gateways, Vald Mirror Gateways can send the request to each other by applying `ValdMirrorTarget`. | ||
|
|
||
| For more information about `ValdMirrorTarget` configuration, please refer to [Custom Resource Configuration](https://vald.vdaas.org/docs/user-guides/mirroring-configuration/). | ||
|
|
||
| ### Request forwarding | ||
|
|
||
| <img src="../../../assets/docs/overview/component/mirror-gateway/request-forwarding.png"> | ||
|
|
||
| The Vald Mirror Gateway forwards the incoming user request ([Insert](https://vald.vdaas.org/docs/api/insert/) / [Upsert](https://vald.vdaas.org/docs/api/upsert/) / [Update](https://vald.vdaas.org/docs/api/update/) / [Remove](https://vald.vdaas.org/docs/api/remove/)) to other Vald Mirror Gateways. | ||
| Then, while forwarding the user request, the Vald Mirror Gateway bypasses the incoming user request to Vald LB Gateway in its own cluster. | ||
|
|
||
| On the other hand, if the incoming user request is an [Object API](https://vald.vdaas.org/docs/api/object/) or [Search API](https://vald.vdaas.org/docs/api/search/), it is bypassed to only a Vald LB Gateway in its own cluster without forwarding it to other Vald Mirror Gateways. | ||
|
|
||
| ### Automatic rollback on failure | ||
|
|
||
| The request may fail at the forwarding destination or the bypass destination. | ||
|
|
||
| If some requests fail, the vector data will not be consistent across Vald clusters. | ||
|
|
||
| To keep index state consistency, the Vald Mirror Gateway will send the rollback request for the failed request. After the rollback request succeeds, the index state will be the same as before requesting. | ||
|
|
||
| The following is the list of rollback types. | ||
|
|
||
| - Insert Request | ||
| - Rollback Condition/Trigger | ||
| - Status code other than `ALREADY_EXISTS` exists | ||
| - Rollback request to the successful request | ||
| - REMOVE request | ||
| - Remove Request | ||
| - Rollback Condition/Trigger | ||
| - Status code other than `NOT_FOUND` exists | ||
| - Rollback request to the successful request | ||
| - UPSERT Request with old vector | ||
| - Update Request | ||
| - Rollback Condition/Trigger | ||
| - Status code other than `ALREADY_EXISTS` exists | ||
| - Rollback request to the successful request | ||
| - REMOVE Request if there is no old vector data | ||
| - UPDATE Request if there is old vector data | ||
| - Upsert Request | ||
| - Rollback Condition/Trigger | ||
| - Status code other than `ALREADY_EXISTS` exists | ||
| - Rollback request to the successful request | ||
| - REMOVE Request if there is no old vector data | ||
| - UPDATE Request if there is old vector data |
Oops, something went wrong.