*: add topic (#480)

* *: add topic

Added topic to Varlog.

* proto: add Topic into proto (#479)

* add topic

* add topic into proto

* wip

* fix CommitContextOf

* add highWatermark into report

* Update internal/metadata_repository/raft_metadata_repository.go

Co-authored-by: Injun Song <ijsong@gmail.com>

* Update internal/metadata_repository/raft_metadata_repository.go

Co-authored-by: Injun Song <ijsong@gmail.com>

* Update internal/metadata_repository/raft_metadata_repository.go

Co-authored-by: Injun Song <ijsong@gmail.com>

* Update internal/metadata_repository/raft_metadata_repository.go

Co-authored-by: Injun Song <ijsong@gmail.com>

* Update internal/metadata_repository/raft_metadata_repository.go

* Update internal/metadata_repository/storage.go

Co-authored-by: Injun Song <ijsong@gmail.com>

* fix code

* fix code

Co-authored-by: Hyungkeun Jang <pharrell.jang@kakaocorp.com>
Co-authored-by: Injun Song <ijsong@gmail.com>

* *: use int32 for storage node id and log stream id (#481)

Changed type of `types.StorageNodeID` and `types.LogStreamID` from
uint32 to int32.

Resolves [#VARLOG-548](VARLOG-548).

* topic: managemant topic (#485)

* add topic

* add topic into proto

* wip

* fix CommitContextOf

* add highWatermark into report

* wip

* management topic

* add test for register topic

* add test for unregister topic

* fmt

* fix code

* fix test

* fix code

* fix code

Co-authored-by: Hyungkeun Jang <pharrell.jang@kakaocorp.com>

* sn: remove redundant types for replica (#483)

There were redundant types to represent replica:

- `pkg/logc/StorageNode`
- `proto/snpb/Replica`
- `proto/snpb/AppendRequest_BackupNode`

This patch removes those and uses `proto/varlogpb/StorageNode` and types
that wrap it.

Resolves [#VARLOG-546](VARLOG-546).

* sn: add topic id to log i/o (#486)

This patch adds `TopicID` to the methods of Log I/O interface. It
doesn't contain any meaningful implementations about `TopicID`.

Types that now have `TopicID` are follows:

- `internal/storagenode/logio.ReadWriter`
- `internal/storagenode/reportcommitter/reportcommitter.Getter`
- `proto/snpb.AppendRequest`, `proto/snpb.AppendResponse`, `proto/snpb.ReadRequest`, `proto/snpb.SubscribeRequest`, `proto/snpb.TrimRequest`
- `proto/varlogpb.Replica`

Resolves [#VARLOG-542](VARLOG-542).

* it: fix flaky test - TestVarlogSubscribeWithAddLS (#487)

The `TestVarlogSubscribeWithAddLS` created a goroutine adding new
 LogStream while appending log entries. It however did not manage the
 life cycle of the goroutine resulting in several issues:

- use closed connection to append logs
- wait for commit messages from MR indefinitely since the MR is already
  closed

This patch simply adds `sync.WaitGroup` to the test to avoid the above
issues.

Resolves [#VARLOG-569](VARLOG-569).

* proto: removing unnecessary fields from messages (#488)

This patch removes unnecessary fields generated automatically from proto
messages. These lines are like follows:

```
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized     []byte   `json:"-"`
XXX_sizecache        int32    `json:"-"`
```

They are something related to either optimizations or compatibility that
are generated by the gogoproto. However, they come to small overhead in
many cases. (See cockroachdb/cockroach#38404 and
etcd-io/etcd@e51c697#diff-76a35072df72591a656e69cab6f6fa99aa386fd5ace35c9042851eb324ec16b5).

This change adds the following options to every proto definition file:

```
option (gogoproto.goproto_unkeyed_all) = false;
option (gogoproto.goproto_unrecognized_all) = false;
option (gogoproto.goproto_sizecache_all) = false;
```

Resolves [#VARLOG-557](VARLOG-557).

* *: dedup LogEntry types (#489)

- Remove `pkg/types.LogEntry`, then use `proto/varlogpb.LogEntry`.

Resolves [#VARLOG-558](VARLOG-558).

* vendor: bump Pebble (#490)

Resolves #VARLOG-556

* sn: rename AddLogStream RPC (#491)

In this patch, the RPC `AddLogStream`  renames to `AddLogStreamReplica`
to clarify its behavior. The RPC `AddLogStreamReplica` adds a new
replica of the given LogStream to the StorageNode.

Resolves [#VARLOG-568](VARLOG-568).

* topic: apply Topic into client (#493)

* logio: apply topic

* add topic test

* fix TestMRTopicLastHighWatermark

* fix managing log stream status in vms

Resolves #VARLOG-559

* all: update golang 1.17.0 (#492)

Resolves [#VARLOG-555](VARLOG-555).

* all: fix code style (#494)

This patch just fixes code styles.

Resolves [#VARLOG-572](VARLOG-572).

* build: use predefined protoc (#496)

Resolves [#VARLOG-563](VARLOG-563).

* sn,topic: checking topic id while handling RPCs (#495)

This patch adds a feature that the StorageNode checks if the topic id is
valid or not. To support this functionality it adds a topicID to a
parameter of many functions.

The executor does not care about the topicID, rather it will be
considered by StorageNode. To do that, StorageNode maintains the
executors by using the executorsmap keyed by logStreamTopicID. The
logStreamTopicID is a packed type of the LogStreamID and TopicID.

Resolves [#VARLOG-542](VARLOG-542).

* lint: fix code style (#497)

This is a follow-up PR for VARLOG-572.

Resolves [#VARLOG-572](VARLOG-572).

Co-authored-by: Hyungkeun Jang <pharrell.jang@kakaocorp.com>
Co-authored-by: Hyungkeun Jang <pharrell.jang@kakaocorp.com>

.gitignore

@@ -33,6 +33,9 @@ bin/vmr
 bin/vms
 bin/vsn
 bin/rpc_test_server
+bin/rpcbench_client
+bin/rpcbench_server
+bin/benchmark
 
 # Python
 *.pyc

.jenkins/Jenkinsfile

@@ -6,49 +6,41 @@ pipeline {
 
     stage("build") {
       steps {
-        sh "make all"
+        sh "make build"
       }
     }
 
     stage("test") {
-      environment { 
-        TEST_USE_LOGGER = "0"
-      }
-
       steps {
-        sh "make test TEST_FAILFAST=1 TEST_TIMEOUT=30m TEST_COUNT=2 TEST_COVERAGE=1"
+        sh "make test_ci TEST_FLAGS='-v -race -failfast -count=2 -timeout=30m'"
       }
 
       post {
         always {
           sh "make test_report"
-          junit "build/reports/*.xml"
+          junit "reports/*.xml"
 
           sh "make coverage_report"
-          cobertura coberturaReportFile: "build/reports/coverage.xml"
+          cobertura coberturaReportFile: "reports/coverage.xml"
         }
       }
     }
 
     /*
     stage("benchmark") {
-      environment {
-        TEST_USE_LOGGER = "0"
-      }
-
       steps {
         sh "make bench"
 
         lock("varlog-sandbox-01") {
-          sh "./scripts/benchmark.sh > build/reports/load.xml"
+          sh "./scripts/benchmark.sh > reports/load.xml"
         }
       }
 
       post {
         always {
           sh "make bench_report"
-          perfReport "build/reports/bench.xml"
-          perfReport "build/reports/load.xml"
+          perfReport "reports/bench.xml"
+          perfReport "reports/load.xml"
         }
       }
     }