Performance Analysis and Improvement of LogStream Primary-Backup Replication

[ijsong]

Background

This issue aims to analyze and improve the performance of LogStream's Primary-Backup Replication. First, we need to define the metrics that accurately reflect replication performance. Currently, there are metrics such as sn.replicate.logs, sn.replicate.bytes, and sn.replicate.duration. However, these metrics only provide average values for replication throughput and response time. These metrics need to be redefined to conduct a detailed analysis using percentiles.

Problem: Current Structure

The replication gRPC operates as follows: the Primary Replica invokes the Backup Replica's Replicate RPC. In this setup, the Backup Replica acts as the gRPC server, while the Primary Replica acts as the client. The Primary Replica sends batches of LogEntry to the Backup Replica using the Replicate RPC.

1. Primary Replica: Sends batches of LogEntry via Replicate RPC.
2. Backup Replica:
   • Receiver Goroutine: This goroutine acts as the gRPC Receiver, receiving ReplicateRequest through Replicate RPC and placing it into a queue as ReplicationRequestTask.
   • Replicator Goroutine: This goroutine takes ReplicationRequestTask from the queue and uses it to call LogStreamExecutor's Replicate method.

The Receiver goroutine defines ReplicateRequest once and uses it continuously. However, the gRPC Unmarshal code repeatedly generates slices for some fields, such as LLSN and Data. Although the protobuf message ReplicateRequest is reused, the slices included are not, causing degraded performance.

varlog/proto/snpb/replicator.pb.go

Lines 69 to 76 in fd1b084

	type ReplicateRequest struct {
	TopicID github_com_kakao_varlog_pkg_types.TopicID `protobuf:"varint,1,opt,name=topic_id,json=topicId,proto3,casttype=github.com/kakao/varlog/pkg/types.TopicID" json:"topic_id,omitempty"`
	LogStreamID github_com_kakao_varlog_pkg_types.LogStreamID `protobuf:"varint,2,opt,name=log_stream_id,json=logStreamId,proto3,casttype=github.com/kakao/varlog/pkg/types.LogStreamID" json:"log_stream_id,omitempty"`
	LLSN []github_com_kakao_varlog_pkg_types.LLSN `protobuf:"varint,3,rep,packed,name=llsn,proto3,casttype=github.com/kakao/varlog/pkg/types.LLSN" json:"llsn,omitempty"`
	Data [][]byte `protobuf:"bytes,4,rep,name=data,proto3" json:"data,omitempty"`
	}
	func (m *ReplicateRequest) Reset() { *m = ReplicateRequest{} }

varlog/internal/storagenode/replication_server.go

Lines 109 to 133 in fd1b084

	func (rs *replicationServer) recv(ctx context.Context, stream snpb.Replicator_ReplicateServer, wg *sync.WaitGroup) <-chan *replicationServerTask {
	wg.Add(1)
	// TODO: add configuration
	c := make(chan *replicationServerTask, 4096)
	go func() {
	defer wg.Done()
	defer close(c)
	req := &snpb.ReplicateRequest{}
	for {
	req.Reset()
	err := stream.RecvMsg(req)
	rst := newReplicationServerTask(*req, err)
	select {
	case c <- rst:
	if err != nil {
	return
	}
	case <-ctx.Done():
	rst.release()
	return
	}
	}
	}()
	return c
	}

Proposed Optimizations

The following ideas are proposed to optimize the replication process:

1. Use sync.Pool for ReplicateRequest: Utilize sync.Pool to reuse ReplicateRequest structures.
2. Reuse Slices: Extend the Reset method generated in gRPC to reuse slice-type fields - LLSN and Data - of the ReplicateRequest struct.

The proposed optimization aims to reuse these fields to enhance Unmarshal's performance. The longer the batch size, the more allocations for the slices there are, so reusing them seems promising. Furthermore, append entries into the slices cause copy overhead whenever the underlying heaps enlarge. This optimization may greatly reduce this overhead.

To reuse the slice in the protobuf message, the following code can be used:

func (m *ReplicateRequest) ResetReuse() {
	s := m.LLSN[:0]
	m.Reset()
	m.LLSN = s
}

In the case of snpb.(ReplicateRequest).Data, the outer slice can be reused using the above code. However, we should change the unmarshalling code generated by gRPC to reuse the inner slice.

Plan

• Define new metrics that include percentiles for detailed analysis. (feat(sn): add metrics for Append and Replicate RPCs #806)
• Utilize sync.Pool to efficiently reuse ReplicateRequest structures. (perf(sn): reuse buffer for ReplicateRequest unmarshaling #808)
• Implement reuse of LLSN and Data fields in ReplicateRequest to reduce overhead. (perf(sn): reuse buffer for ReplicateRequest unmarshaling #808)