CHANGELOG.md

Changelog

0.23.0 (2024-10-23)

Features

• remove gRPC buffer pool flags (282bcaa)

Bug Fixes

• logger: no overriding custom zap logger options (ed416fe)
• metarepos: check the NodeID for adding and removing peers (43003d4)

0.22.1 (2024-07-04)

Bug Fixes

• sn: prevent panic on closed queue channel of replication server (6c19613)

0.22.0 (2024-06-15)

Features

• sn: add metrics for Append and Replicate RPCs (d52ff1d)

0.21.0 (2024-06-11)

Features

• all: add CLI flags for gRPC buffer management (57db00e)
• storage: add CLI flag '--storage-cache-size' for setting cache for storage (e82fd37)

Bug Fixes

• rpc: use google.golang.org/grpc.NewClient instead of DialContext (851f1c9)

0.20.0 (2024-03-05)

Features

• benchmark: use gops to diagnose benchmark tool (9f31816)
• benchmark: use gops to diagnose benchmark tool (#717) (86a8155)

Bug Fixes

• benchmark: refine the benchmark tool for log subscription (01bf188)
• benchmark: refine the benchmark tool for log subscription (#716) (eb83d42)

Performance Improvements

• benchmark: use grpc.SharedBufferPool for grpc.DialOption (8014ece)
• benchmark: use grpc.SharedBufferPool for grpc.DialOption (#718) (fbf0c17)
• client: optimize with atomic.Int64 for lastSubscribeAt in pkg/varlog.(subscriber) (617bd9d)
• client: optimize with atomic.Int64 for lastSubscribeAt in pkg/varlog.(subscriber) (#720) (6d2779e)
• client: prealloc pkg/varlog.(*transmitter).transmitQueue (58a5c21)
• client: prealloc pkg/varlog.(*transmitter).transmitQueue (#721) (26ed1a6)
• client: reuse snpb.SubscribeResponse in RPC handler (890508d)
• client: reuse snpb.SubscribeResponse in RPC handler (#719) (e98ac54)
• storage: enhance GLSN Log scanning efficiency (963d10f)
• storage: improve scanning with GLSN (#715) (3caaa0a)

0.19.2 (2024-02-14)

Bug Fixes

• client: remove subscribe's initial delay (e77c763)

0.19.1 (2024-02-01)

Bug Fixes

• varlogtest: panic on PeekLogStream after removing all logs (203fc33)
• varlogtest: panic on PeekLogStream after removing all logs (#686) (2b2cdfb)

0.19.0 (2024-01-02)

Features

• metarepos: unseal reportCollector with LastCommittedVer - 1 (d29c29a)
• pkg/util/units.FromByteSizeString respects SI and IEC standards (cf0e749), closes #661
• pkg/util/units.FromByteSizeString respects SI and IEC standards (#662) (7e6a2f5), closes #661
• storage: handle inconsistency between data and commit in ReadRecoveryPoints (8832c06)
• storage: handle inconsistency between data and commit in ReadRecoveryPoints (#545) (208c113)

0.18.1 (2023-10-17)

Bug Fixes

• varlogsn: fix JSON parsing of get_data_dirs in start_varlogsn (543d551)

0.18.0 (2023-10-10)

Features

• metarepos: rename the executable from vmr to varlogmr (0ef2d85)
• metarepos: rename the executable from vmr to varlogmr (#610) (52de516)

Bug Fixes

• e2e: run the correct number of MR nodes (b1e8432)
• e2e: run the correct number of MR nodes (#609) (89a94e3)
• metarepos: use --replication-factor to set the replication factor (ef29008)
• metarepos: use --replication-factor to set the replication factor (#608) (1bdc345)

0.17.0 (2023-10-10)

Features

• all: add version to the daemons (ab345de)
• all: add version to the daemons (#598) (a430609)
• varlogtest: support ListLogStreams API (c0c2e9d)
• varlogtest: support ListLogStreams API (#607) (1f42351)

0.16.0 (2023-10-04)

Features

• admin: check cluster id in mrmanager (b958fe0)
• admin: check cluster id in mrmanager (#555) (7794a6d)
• admin: check replication factor in mrmanager (a18b285)
• admin: check replication factor in mrmanager (#556) (2a1b5b6)
• all: add gogoproto codec to grpc servers (5e1f344)
• all: add gogoproto codec to grpc servers (#581) (6127cd2)
• all: change type of ClusterID from uint32 to int32 (500a4bb)
• all: change type of ClusterID from uint32 to int32 (#566) (ac89e40)
• varlogtest: add functional options to varlogtest (97dab98)
• varlogtest: add functional options to varlogtest (#582) (eb8a96f)
• varlogtest: add initial metadata repository and support fetch APIs (c033d62)
• varlogtest: add initial metadata repository and support fetch APIs (#583) (626bdf8)
• varlogtest: create new instance of admin and log (2c98f7f)
• varlogtest: create new instance of admin and log (#591) (385446d)

Bug Fixes

• admin: refresh cluster metadata while adding storage node (a2e0326)
• admin: refresh cluster metadata while adding storage node (#558) (4cb06f1)
• rpc: call Reset before unmarshaling protobuf (aae2495)
• varlogtest: log stream appender can be closed in the callback (e08371b)
• varlogtest: log stream appender can be closed in the callback (#595) (2480922)
• varlogtest: log stream appender returns missing errors in the callback (9d0b203)
• varlogtest: log stream appender returns missing errors in the callback (#594) (d2e88ee)
• varlogtest: validate log stream descriptor to create it (73d6a91)
• varlogtest: validate log stream descriptor to create it (#593) (27cd3bb)

Performance Improvements

• rpc: use gogoproto codec (#236) (c4224c0)

0.15.0 (2023-07-31)

Features

• admin: add otelgrpc metric interceptor (d9ca9aa)
• admin: add otelgrpc metric interceptor (#509) (db7a1a2)
• admin: speed up fetching cluster metadata (3e46f62)
• admin: speed up fetching cluster metadata (#480) (53a8f19)
• all: add common flags for telemetry (fcacd1a)
• all: add common flags for telemetry (#494) (63355e9)
• benchmark: share a connection between appenders in a target (7dc53e9)
• benchmark: share a connection between appenders in a target (#524) (2cd9196)
• client: add Clear to the log stream appender manager (9a89065)
• client: add Clear to the log stream appender manager (#514) (e5b6a2e)
• storagenode: add --storage-trim-delay to set a delay before the deletion of log entries (db39713)
• storagenode: add --storage-trim-delay to set a delay before the deletion of log entries (#529) (015bfa4)
• storagenode: add --storage-trim-rate to set throttling rate of Trim (83b7496)
• storagenode: add --storage-trim-rate to set throttling rate of Trim (#530) (6e69306)
• telemetry: customize bucket size of process.runtime.go.gc.pause_ns (b181132)
• telemetry: customize bucket size of process.runtime.go.gc.pause_ns (#510) (9d99520)
• telemetry: customize bucket size of rpc.server.duration (a0e5973)
• telemetry: customize bucket size of rpc.server.duration (#511) (e41fe1c)

Bug Fixes

• benchmark: make append duration's precision high (e3a091d)
• benchmark: make append duration's precision high (#522) (815af53)
• benchmark: support graceful stop (8616d55)
• benchmark: support graceful stop (#527) (fc4ed81)
• metarepos: add TestMRIgnoreDirtyReport (fe2a550)
• metarepos: allow set commitTick (bdca20a)
• metarepos: ignore invalid report (e8620de)
• storagenode: ignore context error while checking to interleave of Append RPC errors (04d1052)
• storagenode: ignore context error while checking to interleave of Append RPC errors (#504) (5a7a3b0)
• storagenode: restore uncommitted logs (267cccc), closes #490
• storagenode: restore uncommitted logs (#492) (a9832ee), closes #490

Performance Improvements

• admin: use singleflight to handle Admin's RPCs (c231888)
• admin: use singleflight to handle Admin's RPCs (#482) (1a6a96d)
• metarepos: add a pool for []*mrpb.Report (fa8c89d)
• metarepos: add a pool for []*mrpb.Report (#534) (16b2181)
• metarepos: add a pool for *mrpb.RaftEntry (be9f121)
• metarepos: add a pool for *mrpb.RaftEntry (#536) (96ab5e2)
• metarepos: add a pool for mrpb.Reports (59a6a5a)
• metarepos: add a pool for mrpb.Reports (#533) (b227c75)
• metarepos: avoid copy overhead by removing unnecessary converting from byte slice to string (a775628)
• metarepos: avoid copy overhead by removing unnecessary converting from byte slice to string (#532) (1702769)
• metarepos: reuse mrpb.StorageNodeUncommitReport while changed (57d8039)
• metarepos: reuse mrpb.StorageNodeUncommitReport while changed (#537) (8f6e097)

0.14.1 (2023-06-20)

Bug Fixes

• client: add missing method pkg/varlog.(*MockLog).AppendableLogStreams (7bf9bf9)
• client: add missing method pkg/varlog.(*MockLog).AppendableLogStreams (#487) (61747ed)

0.14.0 (2023-06-19)

Features

• client: add call timeout to log stream appender (6f916d4)
• client: add call timeout to log stream appender (#474) (6db401a)
• client: add LogStreamAppender (dec3421)
• client: add LogStreamAppender (#459) (bfe88d7), closes #433
• client: add settings for gRPC client (917e5fc)
• client: add settings for gRPC client (#479) (4b8f01c)
• client: can call Close at callback in LogStreamAppender (77b8de8)
• client: can call Close at callback in LogStreamAppender (#473) (23236c1)
• client: log stream appender manager (64eae0b)
• client: log stream appender manager (#475) (ac6c7c6)
• client: support goroutine-safety of pkg/var log.(LogStream Appender).Append Batch (7fa53f3)
• client: support goroutine-safety of pkg/var log.(LogStream Appender).Append Batch (#472) (46f0d0b)
• storagenode: add configurations for initial window size (8b623b0)
• storagenode: add configurations for initial window size (#451) (5269481)
• storagenode: add pipelined Append RPC handler (8535a4d)
• storagenode: add pipelined Append RPC handler (#457) (0323629)
• storagenode: change append rpc from unary to stream (5b27a18)
• storagenode: change append rpc from unary to stream (#449) (7e4ef03)

Bug Fixes

• storagenode: accept SyncInit sent from trimmed source to new destination (0e141bd), closes #478
• storagenode: support partial success/failure for append (5d438bb)
• storagenode: support partial success/failure for append (#450) (c8b7fe0)

0.13.0 (2023-06-01)

Features

• admin: add a new flag replica-selector to the varlogadm (805f8de), closes #393
• all: add flags for logger (7efe407), closes #439
• all: add flags for logger (#447) (f2e1193), closes #439
• storage: separate storage databases experimentally (b3845f5)
• storage: separate storage databases experimentally (#410) (9f64785)

Bug Fixes

• metarepos: new topic should start from MinGLSN (d1ae8c8)

Performance Improvements

• metarepos: add mrpb.StorageNodeUncommitReport pool (4c624da)
• metarepos: add mrpb.StorageNodeUncommitReport pool (#446) (60f2cfe)
• storagenode: check log level (39cdbae)
• storagenode: check log level (#411) (da1409d)
• storagenode: estimate the number of batchlets (3c91b62)
• storagenode: estimate the number of batchlets (#414) (cb25d19)
• storagenode: remove backup from append request (f75ef55)
• storagenode: remove backup from append request (#412) (0abcb1a)
• storagenode: wrap replicateTask slice with struct (37250e1), closes #75
• storagenode: wrap replicateTask slice with struct (#416) (911b5fe), closes #75

0.12.0 (2023-04-06)

Features

• admin: add a name to the replica selector (6e7e74f), closes #393
• admin: LFU replica selector (0e08901), closes #393
• admin: random replica selector (ad1226e), closes #393
• storage: metrics logging and less verbose event logging (3b47f8b)
• storagenode: add automaxprocs to storagenode (64f2afe)
• storagenode: change SyncInit to handle trimmed source replica (79c5323), closes #351
• storagenode: restore trimmed log stream replica (02106fa), closes #351
• storagenode: trim API with no safety gap (2fe4a8e), closes #351

Bug Fixes

• metarepos: fix race in TestReportCommit (ff21f66), closes #379
• storagenode: configure ballast by using flags (cc38fcc)
• storagenode: set the flag --storage-mem-table-size correctly (872d796)
• storagenode: subscribe empty range below the global high watermark (1ef39f8), closes #375
• storage: prints storage log clearly (6e868be)

Performance Improvements

• storage: change storage key prefixes (8d49523)

0.11.0 (2023-02-20)

Features

• admin: defines error codes of several RPCs in the admin server (f5ed66f), closes #312
• metarepos: add grpc error codes to the metadata repository service (2903f8c)
• metarepos: fix grpc error code (ce5feb3)
• storagenode: add gRPC error codes to the admin service (80cd082), closes #312
• storagenode: add gRPC error codes to the log server (5e813fc), closes #312
• varlogcli: do not deny filtered logsteam (26fec68)
• varlogcli: select log stream within AllowedLogStreams (26627a0)

0.10.0 (2023-01-04)

Features

• admin: return ResourceExhausted if the log streams count overflows (b15f29e)
• admin: Updates are rejected if there is no sealed replica (25cef3d)
• admin: Updates are rejected if there is no sealed replica (eef3b9f)
• metarepos: add an upper limit for the number of log streams in a topic (ad2a60f), closes #297
• metarepos: add an upper limit for the number of topics in a cluster (77c6ee4), closes #295
• metarepos: Prevent log loss due to UpdateLogStream (c319333)
• storagenode: add a new sync state - SyncStateStart (b44fd55), closes #299
• storagenode: add an upper limit of log stream replicas count in a storage node (2cfc8bf), closes #293

Bug Fixes

• benchmark: consider multi-target workloads in benchmark webapp (56f338e)

0.9.2 (2022-12-16)

Bug Fixes

• benchmark: fix sql for target and workload (cdc1268)

0.9.1 (2022-12-16)

Bug Fixes

• benchmark: create a new row and get the row from the tables (760aef4)

0.9.0 (2022-12-14)

Features

• benchmark: add --print-json to print benchmark result as JSON (abf8a5a), closes #257
• benchmark: add save command to benchmark (25ecb80), closes #257
• benchmark: add a test command to the benchmark tool (0e08249), closes #257
• benchmark: add initdb command to initialize benchmark database (9d115eb), closes #264
• client,storagenode: remove Head and Tail from proto/varlogpb.(LogStreamDescriptor) (57161c8), closes #73

Bug Fixes

• storagenode: fix concurrency bugs of settings for storage and executor (fdd1781), closes #262

0.8.1 (2022-12-07)

Bug Fixes

• storagenode,client: add missing mock files (436f0d9)

0.8.0 (2022-12-06)

Features

• client: add PeekLogStream to the client (e872677), closes #239
• storagenode: add SyncReplicateStream to synchronize replicas by using stream gRPC (d8d7888), closes #241

0.7.1 (2022-11-23)

Bug Fixes

• dockerfile: use go 1.19 (923f35f)

0.7.0 (2022-11-22)

Features

• benchmark: rework benchmark (9c3f84a), closes #209
• define CommitBatchResponse (6046c99)
• storage: introduce append batch (0534ad0), closes #125
• storagenode: change the synchronization method to accept only the last commit context (8d331f6), closes #125
• storagenode: make log stream executor sealing when reportCommitBase is invalid (6f90720), closes #125
• storagenode: remove bad data dirs and deprecate --data-dirs and --volume-strict-check (1972c94), closes #215
• unify uncommittedLLSNBegin and localHighWatermark in logStreamContext (de88bcf)

Performance Improvements

• storage: use a lightweight method to get the data size of the storage (0965fd9), closes #210

0.6.0 (2022-10-12)

Features

• client: deny lsid if not allowed (c8ad568)

0.5.0 (2022-10-07)

Features

• client: add allowed logstream list option to varlog client (1ce80e7)
• metarepos: ignore invalid report (de83a08)
• storage: change the trim not to remove the commit context (b80964b), closes #125
• storagenode: replica in the learning state does not report to a metadata repository (ca4c184), closes #125
• storagenode: restore the status of a log stream replica by using the latest commit context (9e042d2), closes #125
• storagenode: store only the latest commit context for every commit (ecf3a12), closes #125

Bug Fixes

• storagenode: fix error-prone state management of SyncInit and Report (c7366c5)
• storagenode: remove data directory in removing log stream replica (27fb13f), closes #157

0.4.1 (2022-09-13)

Bug Fixes

• storagenode: do not log payload of /varlog.snpb.Management/AddLogStreamReplca (2cdb74a)

0.4.0 (2022-09-12)

Features

• add proto/varlogpb.(ReplicaDescriptor).DataPath (4db1605), closes #124

Bug Fixes

• admin: succeed to /varlog.vmspb.ClusterManager/UpdateLogStream when already updated (3c451a1), closes #126
• storagenode: fix a response of SyncInit when source replica was trimmed (72e152a), closes #134

Performance Improvements

• storagenode: load log streams concurrently (a59068e), closes #138

0.3.1 (2022-09-06)

Bug Fixes

• admin: add log stream replicas to result of ListStorageNodes (9bce94c), closes #106
• storagenode: make volumes absolute (edfd550), closes #116

0.3.0 (2022-09-04)

Features

• admin: use gRPC codes to clarify errors returned from proto/vmspb.UpdateLogStream RPC (8f539e1), closes #107

Bug Fixes

• admin: contains all storage nodes registered to the cluster (28c02ab), closes #106

0.2.3 (2022-08-29)

Bug Fixes

• storage: apply WithoutSync option correctly (2c25007), closes #102

0.2.2 (2022-08-29)

Bug Fixes

• add error handling to WalkFunc in filepath.Walk in pkg/util/fputil.DirectorySize (ad4e5d8)
• set empty list to logStreams when it is null (0ee5627), closes #88

0.2.1 (2022-08-29)

Bug Fixes

• admin: set empty slice to logStreams in GetStorageNode response rather than null when no log stream replicas (200511e), closes #88
• mr: handle duplicated RegisterLogStream(#62) (861b3b0)

0.2.0 (2022-08-25)

Features

• sn: add CLI flags to varlogsn (85c7f11)

0.1.0 (2022-08-25)

Features

• admin: add --loglevel flag to varlogadm (77bfdf3), closes #79

0.0.3 (2022-08-25)

Bug Fixes

• mr: let newbie logstream know cur version (cd12789)

0.0.2 (2022-08-17)

Bug Fixes

• admin: add handler timeout for failed sn (a2f31d7), closes #29
• remove mutex in storage node manager of admin (77ed718), closes #30

0.0.1 (2022-08-14)

Bug Fixes

• TestAdmin_GetStorageNode_FailedStorageNode (#13) (5c8a3c2)

Miscellaneous Chores

• release 0.0.1 (#21) (6aad0d8)