Skip to content
This repository has been archived by the owner on Dec 10, 2021. It is now read-only.

Commit

Permalink
Add cluster manager (#48)
Browse files Browse the repository at this point in the history
  • Loading branch information
@mosuka
mosuka authored Mar 28, 2019
1 parent 4595341 commit 1a47026
Show file tree
Hide file tree
Showing 173 changed files with 80,120 additions and 101 deletions.
85 changes: 72 additions & 13 deletions Gopkg.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 15 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ You can see the binary file when build successful like so:

```bash
$ ls ./bin
blast-index
blast-indexer
```


Expand Down Expand Up @@ -235,7 +235,7 @@ $ make \
Running a Blast index node is easy. Start Blast data node like so:

```bash
$ ./bin/blast-index start --node-id=index1 --data-dir=/tmp/blast/index1 --bind-addr=:6060 --grpc-addr=:5050 --http-addr=:8080 --index-mapping-file=./example/index_mapping.json
$ ./bin/blast-indexer start --node-id=index1 --data-dir=/tmp/blast/index1 --bind-addr=:6060 --grpc-addr=:5050 --http-addr=:8080 --index-mapping-file=./example/index_mapping.json
```

Please refer to following document for details of index mapping:
Expand All @@ -252,7 +252,7 @@ You can now put, get, search and delete the documents via CLI.
For document indexing, execute the following command:

```bash
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-index index --grpc-addr=:5050 --id=enwiki_1
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-indexer index --grpc-addr=:5050 --id=enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -269,7 +269,7 @@ You can see the result in JSON format. The result of the above command is:
Getting a document is as following:

```bash
$ ./bin/blast-index get --grpc-addr=:5050 --id=enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5050 --id=enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -290,7 +290,7 @@ You can see the result in JSON format. The result of the above command is:
Searching documents is as like following:

```bash
$ cat ./example/search_request.json | xargs -0 ./bin/blast-index search --grpc-addr=:5050
$ cat ./example/search_request.json | xargs -0 ./bin/blast-indexer search --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -462,7 +462,7 @@ Please refer to following document for details of search request and result:
Deleting a document is as following:

```bash
$ ./bin/blast-index delete --grpc-addr=:5050 --id=enwiki_1
$ ./bin/blast-indexer delete --grpc-addr=:5050 --id=enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -479,7 +479,7 @@ You can see the result in JSON format. The result of the above command is:
Indexing documents in bulk, run the following command:

```bash
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-index index --grpc-addr=:5050
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-indexer index --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -496,7 +496,7 @@ You can see the result in JSON format. The result of the above command is:
Deleting documents in bulk, run the following command:

```bash
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-index delete --grpc-addr=:5050
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-indexer delete --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -572,8 +572,8 @@ $ curl -X DELETE 'http://127.0.0.1:8080/documents' -d @./example/docs_wiki.json
Blast is easy to bring up the cluster. Blast data node is already running, but that is not fault tolerant. If you need to increase the fault tolerance, bring up 2 more data nodes like so:

```bash
$ ./bin/blast-index start --node-id=index2 --data-dir=/tmp/blast/index2 --bind-addr=:6061 --grpc-addr=:5051 --http-addr=:8081 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
$ ./bin/blast-index start --node-id=index3 --data-dir=/tmp/blast/index3 --bind-addr=:6062 --grpc-addr=:5052 --http-addr=:8082 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
$ ./bin/blast-indexer start --node-id=index2 --data-dir=/tmp/blast/index2 --bind-addr=:6061 --grpc-addr=:5051 --http-addr=:8081 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
$ ./bin/blast-indexer start --node-id=index3 --data-dir=/tmp/blast/index3 --bind-addr=:6062 --grpc-addr=:5052 --http-addr=:8082 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
```

_Above example shows each Blast node running on the same host, so each node must listen on different ports. This would not be necessary if each node ran on a different host._
Expand All @@ -582,7 +582,7 @@ This instructs each new node to join an existing node, each node recognizes the
So you have a 3-node cluster. That way you can tolerate the failure of 1 node. You can check the peers with the following command:

```bash
$ ./bin/blast-index cluster --grpc-addr=:5050
$ ./bin/blast-indexer cluster --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -621,13 +621,13 @@ Recommend 3 or more odd number of nodes in the cluster. In failure scenarios, da
The following command indexes documents to any node in the cluster:

```bash
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-index index --grpc-addr=:5050 enwiki_1
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-indexer index --grpc-addr=:5050 enwiki_1
```

So, you can get the document from the node specified by the above command as follows:

```bash
$ ./bin/blast-index get --grpc-addr=:5050 enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5050 enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -645,8 +645,8 @@ You can see the result in JSON format. The result of the above command is:
You can also get the same document from other nodes in the cluster as follows:

```bash
$ ./bin/blast-index get --grpc-addr=:5051 enwiki_1
$ ./bin/blast-index get --grpc-addr=:5052 enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5051 enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5052 enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/cluster.go → cmd/blast-indexer/cluster.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ import (
"fmt"
"os"

"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
"github.com/urfave/cli"
)

func execCluster(c *cli.Context) error {
grpcAddr := c.String("grpc-addr")

client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/delete.go → cmd/blast-indexer/delete.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"fmt"
"os"

"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
pbindex "github.com/mosuka/blast/protobuf/index"
"github.com/urfave/cli"
)
Expand Down Expand Up @@ -64,7 +64,7 @@ func execDelete(c *cli.Context) error {
}

// create client
client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/get.go → cmd/blast-indexer/get.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"fmt"
"os"

"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
"github.com/mosuka/blast/protobuf"
pbindex "github.com/mosuka/blast/protobuf/index"
"github.com/urfave/cli"
Expand All @@ -38,7 +38,7 @@ func execGet(c *cli.Context) error {
Id: id,
}

client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/index.go → cmd/blast-indexer/index.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import (
"os"

"github.com/golang/protobuf/ptypes/any"
"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
"github.com/mosuka/blast/protobuf"
pbindex "github.com/mosuka/blast/protobuf/index"
"github.com/urfave/cli"
Expand Down Expand Up @@ -93,7 +93,7 @@ func execIndex(c *cli.Context) error {
}

// create gRPC client
client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
Loading

0 comments on commit 1a47026

Please sign in to comment.