Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove vcache for vald agent due to vcache delete timing control failure and time ordered concurrent vector queue called vqueue #1028

Merged
merged 11 commits into from
Mar 3, 2021
Merged

remove vcache for vald agent due to vcache delete timing control failure and time ordered concurrent vector queue called vqueue #1028

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f4a5911
remove vcache for vald agent due to vcache delete timing control fail…
kpango Feb 27, 2021
2fca4aa
add buffer size config params & add comment for struct fields
kpango Mar 1, 2021
82f0c36
add queue duplicated removal logic comment
kpango Mar 1, 2021
70ec88f
change VCacheLen method name to VQueueLen and change metrics name vca…
kpango Mar 1, 2021
f9dc344
fix failed test
kpango Mar 1, 2021
428bd6a
Merge branch 'master' into bugfix/agent/remove-vcache-due-to-timing-c…
kpango Mar 1, 2021
160fa3a
add channel buffer length metrics
kpango Mar 1, 2021
227d472
Merge branch 'master' into bugfix/agent/remove-vcache-due-to-timing-c…
kpango Mar 2, 2021
ec0b1e6
add delete queue buffer to uncommited index response
kpango Mar 3, 2021
facbd8d
replace go-hostpool org monzo to kpango as a temporary workaround
kpango Mar 3, 2021
78997d6
bugfix/agent/remove-vcache-due-to-timing-control-failure-and-add-conc…
kpango Mar 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/vald/values.schema.json
View file
Open in desktop

Large diffs are not rendered by default.

14 changes: 14 additions & 0 deletions charts/vald/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2578,6 +2578,20 @@ agent:
# @schema {"name": "agent.ngt.enable_proactive_gc", "type": "boolean"}
# agent.ngt.enable_proactive_gc -- enable proactive GC call for reducing heap memory allocation
enable_proactive_gc: true
# @schema {"name": "agent.ngt.vqueue", "type": "object"}
vqueue:
# @schema {"name": "agent.ngt.vqueue.insert_buffer_size", "type": "integer"}
# agent.ngt.vqueue.insert_buffer_size -- insert channel buffer size
insert_buffer_size: 100
# @schema {"name": "agent.ngt.vqueue.delete_buffer_size", "type": "integer"}
# agent.ngt.vqueue.delete_buffer_size -- delete channel buffer size
delete_buffer_size: 100
# @schema {"name": "agent.ngt.vqueue.insert_buffer_pool_size", "type": "integer"}
# agent.ngt.vqueue.insert_buffer_pool_size -- insert slice pool buffer size
insert_buffer_pool_size: 10000
# @schema {"name": "agent.ngt.vqueue.delete_buffer_pool_size", "type": "integer"}
# agent.ngt.vqueue.delete_buffer_pool_size -- delete slice pool buffer size
delete_buffer_pool_size: 5000
# @schema {"name": "agent.sidecar", "type": "object"}
sidecar:
# @schema {"name": "agent.sidecar.enabled", "type": "boolean"}
Expand Down
4 changes: 2 additions & 2 deletions cmd/agent/core/ngt/sample.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,9 @@ ngt:
bulk_insert_chunk_size: 10
creation_edge_size: 20
default_epsilon: 0.01
default_pool_size: 10000
default_pool_size: 100
default_radius: -1
dimension: 6
dimension: 100
distance_type: l2
enable_in_memory_mode: true
enable_proactive_gc: true
Expand Down
14 changes: 7 additions & 7 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,23 @@ go 1.16
replace (
cloud.google.com/go => cloud.google.com/go v0.78.0
github.com/Azure/go-autorest => github.com/Azure/go-autorest v14.2.0+incompatible
github.com/aws/aws-sdk-go => github.com/aws/aws-sdk-go v1.37.20
github.com/aws/aws-sdk-go => github.com/aws/aws-sdk-go v1.37.22
github.com/boltdb/bolt => github.com/boltdb/bolt v1.3.1
github.com/chzyer/logex => github.com/chzyer/logex v1.1.11-0.20170329064859-445be9e134b2
github.com/coreos/etcd => go.etcd.io/etcd v3.3.25+incompatible
github.com/docker/docker => github.com/moby/moby v20.10.4+incompatible
github.com/docker/docker => github.com/moby/moby v20.10.5+incompatible
github.com/envoyproxy/protoc-gen-validate => github.com/envoyproxy/protoc-gen-validate v0.4.1
github.com/go-sql-driver/mysql => github.com/go-sql-driver/mysql v1.5.0
github.com/gocql/gocql => github.com/gocql/gocql v0.0.0-20210129204804-4364a4b9cfdd
github.com/gocql/gocql => github.com/gocql/gocql v0.0.0-20210301062520-a04dba85ed25
github.com/gogo/googleapis => github.com/gogo/googleapis v1.4.0
github.com/gogo/protobuf => github.com/gogo/protobuf v1.3.2
github.com/google/go-cmp => github.com/google/go-cmp v0.5.4
github.com/google/pprof => github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5
github.com/googleapis/gnostic => github.com/googleapis/gnostic v0.5.4
github.com/gophercloud/gophercloud => github.com/gophercloud/gophercloud v0.16.0
github.com/gorilla/websocket => github.com/gorilla/websocket v1.4.2
github.com/hailocab/go-hostpool => github.com/monzo/go-hostpool v0.0.0-20200724120130-287edbb29340
github.com/klauspost/compress => github.com/klauspost/compress v1.11.9-0.20210226154345-6381575b273f
github.com/hailocab/go-hostpool => github.com/kpango/go-hostpool v0.0.0-20210303030322-aab80263dcd0
github.com/klauspost/compress => github.com/klauspost/compress v1.11.10-0.20210302113407-0d78ef07f573
github.com/kpango/glg => github.com/kpango/glg v1.5.1
github.com/tensorflow/tensorflow => github.com/tensorflow/tensorflow v2.1.2+incompatible
golang.org/x/crypto => golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
Expand Down Expand Up @@ -76,11 +76,11 @@ require (
go.uber.org/zap v1.16.0
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/sys v0.0.0-20210227040730-b0d1d43c014d
golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b
gonum.org/v1/hdf5 v0.0.0-20200504100616-496fefe91614
gonum.org/v1/plot v0.8.1
google.golang.org/api v0.40.0
google.golang.org/genproto v0.0.0-20210226172003-ab064af71705
google.golang.org/genproto v0.0.0-20210302174412-5ede27ff9881
google.golang.org/grpc v1.35.0
gopkg.in/yaml.v2 v2.4.0
k8s.io/api v0.20.4
Expand Down
24 changes: 12 additions & 12 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hC
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/aws/aws-sdk-go v1.37.20 h1:CJCXpMYmBJrRH8YwoSE0oB9S3J5ax+62F14sYlDCztg=
github.com/aws/aws-sdk-go v1.37.20/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
github.com/aws/aws-sdk-go v1.37.22 h1:cyZp8TvUbH9rrShdrwULtCj4pB5szddrw9aKHUsw1Ic=
github.com/aws/aws-sdk-go v1.37.22/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
Expand Down Expand Up @@ -169,8 +169,8 @@ github.com/go-redis/redis/v8 v8.6.0/go.mod h1:DQ9q4Rk2HtwkrwVrdgmphoOQDMfpvcd/nH
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gocql/gocql v0.0.0-20210129204804-4364a4b9cfdd h1:j0VZcDIpU6sLBMNFUEvuwP4FdmEXbmaEq6qufApwbbo=
github.com/gocql/gocql v0.0.0-20210129204804-4364a4b9cfdd/go.mod h1:DL0ekTmBSTdlNF25Orwt/JMzqIq3EJ4MVa/J/uK64OY=
github.com/gocql/gocql v0.0.0-20210301062520-a04dba85ed25 h1:RqyFvP1RF5WLMGn9VVCSTPzg15b7jvFwJeWmWujSA6Q=
github.com/gocql/gocql v0.0.0-20210301062520-a04dba85ed25/go.mod h1:DL0ekTmBSTdlNF25Orwt/JMzqIq3EJ4MVa/J/uK64OY=
github.com/gocraft/dbr/v2 v2.7.1 h1:v/bqxEBTlzj9sDbFCmtO2K1M3foDnYtQn097VoMALCw=
github.com/gocraft/dbr/v2 v2.7.1/go.mod h1:+q/T5TMzrc6BWFbcSzj7ixaaBJkbDfRsy9gq7eMsU1s=
github.com/gogo/googleapis v1.4.0 h1:zgVt4UpGxcqVOw97aRGxT4svlcmdK35fynLNctY32zI=
Expand Down Expand Up @@ -295,8 +295,8 @@ github.com/jung-kurt/gofpdf v1.16.2 h1:jgbatWHfRlPYiK85qgevsZTHviWXKwB1TTiKdz5Pt
github.com/jung-kurt/gofpdf v1.16.2/go.mod h1:1hl7y57EsiPAkLbOwzpzqgx1A30nQCk/YmFV8S2vmK0=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.11.9-0.20210226154345-6381575b273f h1:Ci0TKv+M1ZYVcpHxft0R0GR2RQ6Y6v32JJ4FKmkKo38=
github.com/klauspost/compress v1.11.9-0.20210226154345-6381575b273f/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/compress v1.11.10-0.20210302113407-0d78ef07f573 h1:ITN78bwwn3Ie6wqQ7Xq/dSsG8cJIj0+ryIh0sq2ETJI=
github.com/klauspost/compress v1.11.10-0.20210302113407-0d78ef07f573/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
Expand All @@ -309,6 +309,8 @@ github.com/kpango/gache v1.2.5 h1:JBR4k74TSRWrfK7zk4godTl3lS4X1BWhdVD645NQYQ4=
github.com/kpango/gache v1.2.5/go.mod h1:xiTyKBFCsq/KA3fu4aznKwnnmfMLfOch5ZI/gAD2iPY=
github.com/kpango/glg v1.5.1 h1:ecOOgdPMt7OdDUYjoUZ9dbnY8MVwUUMc6D5ZN3exLNM=
github.com/kpango/glg v1.5.1/go.mod h1:xIbZZSoRgDaYrXYmBK4wccGHkHK3qk61H/pK3R4qyE8=
github.com/kpango/go-hostpool v0.0.0-20210303030322-aab80263dcd0 h1:orIEVdc68woWO1ZyYWEVOl5Kl33eDjP+kbxgbdpMwi4=
github.com/kpango/go-hostpool v0.0.0-20210303030322-aab80263dcd0/go.mod h1:CJt2OTBmIWH7X/gYDy8fg67d26E/a8MpbGvgrKWZQxQ=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
Expand Down Expand Up @@ -358,8 +360,6 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJ
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/monzo/go-hostpool v0.0.0-20200724120130-287edbb29340 h1:QuK697dDaOR2VjdQTtMa3kRAWdxr/upvblGXmCxrg5k=
github.com/monzo/go-hostpool v0.0.0-20200724120130-287edbb29340/go.mod h1:XXO50svJbofYOBxMA6c3sQMMQ71W/kV5AjMZgjxUN3s=
github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
Expand Down Expand Up @@ -686,8 +686,8 @@ golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210220050731-9a76102bfb43/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210227040730-b0d1d43c014d h1:9fH9JvLNoSpsDWcXJ4dSE3lZW99Z3OCUZLr07g60U6o=
golang.org/x/sys v0.0.0-20210227040730-b0d1d43c014d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b h1:kHlr0tATeLRMEiZJu5CknOw/E8V6h69sXXQFGoPtjcc=
golang.org/x/sys v0.0.0-20210301091718-77cc2087c03b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
Expand Down Expand Up @@ -814,8 +814,8 @@ google.golang.org/genproto v0.0.0-20201019141844-1ed22bb0c154/go.mod h1:FWY/as6D
google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210222152913-aa3ee6e6a81c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210226172003-ab064af71705 h1:PYBmACG+YEv8uQPW0r1kJj8tR+gkF0UWq7iFdUezwEw=
google.golang.org/genproto v0.0.0-20210226172003-ab064af71705/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210302174412-5ede27ff9881 h1:SYuy3hIRsBIROE0aZwsJZOEJNC/n9/p0FmLEU9C31AE=
google.golang.org/genproto v0.0.0-20210302174412-5ede27ff9881/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/grpc v1.36.0 h1:o1bcQ6imQMIOpdrO3SWf2z5RV72WbDwdXuK0MDlc8As=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
Expand Down
2 changes: 1 addition & 1 deletion hack/go.mod.default
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ replace (
github.com/googleapis/gnostic => github.com/googleapis/gnostic latest
github.com/gophercloud/gophercloud => github.com/gophercloud/gophercloud latest
github.com/gorilla/websocket => github.com/gorilla/websocket latest
github.com/hailocab/go-hostpool => github.com/monzo/go-hostpool latest
github.com/hailocab/go-hostpool => github.com/kpango/go-hostpool master
github.com/klauspost/compress => github.com/klauspost/compress master
github.com/tensorflow/tensorflow => github.com/tensorflow/tensorflow v2.1.2
golang.org/x/crypto => golang.org/x/crypto latest
Expand Down
4 changes: 2 additions & 2 deletions internal/config/net_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ func TestDialer_Bind(t *testing.T) {
}
}

func TestTCP_Bind(t *testing.T) {
func TestNet_Bind(t *testing.T) {
type fields struct {
DNS *DNS
Dialer *Dialer
Expand Down Expand Up @@ -267,7 +267,7 @@ func TestTCP_Bind(t *testing.T) {
}
}

func TestTCP_Opts(t *testing.T) {
func TestNet_Opts(t *testing.T) {
type fields struct {
DNS *DNS
Dialer *Dialer
Expand Down
61 changes: 41 additions & 20 deletions internal/config/ngt.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,64 +20,82 @@ package config
// NGT represent the ngt core configuration for server.
type NGT struct {
// IndexPath represent the ngt index file path
IndexPath string `yaml:"index_path" json:"index_path"`
IndexPath string `yaml:"index_path" json:"index_path,omitempty"`

// Dimension represent the ngt index dimension
Dimension int `yaml:"dimension" json:"dimension"`
Dimension int `yaml:"dimension" json:"dimension,omitempty"`

// BulkInsertChunkSize represent the bulk insert chunk size
BulkInsertChunkSize int `yaml:"bulk_insert_chunk_size" json:"bulk_insert_chunk_size"`
BulkInsertChunkSize int `yaml:"bulk_insert_chunk_size" json:"bulk_insert_chunk_size,omitempty"`

// DistanceType represent the ngt index distance type
DistanceType string `yaml:"distance_type" json:"distance_type"`
DistanceType string `yaml:"distance_type" json:"distance_type,omitempty"`

// ObjectType represent the ngt index object type float or int
ObjectType string `yaml:"object_type" json:"object_type"`
ObjectType string `yaml:"object_type" json:"object_type,omitempty"`

// CreationEdgeSize represent the index edge count
CreationEdgeSize int `yaml:"creation_edge_size" json:"creation_edge_size"`
CreationEdgeSize int `yaml:"creation_edge_size" json:"creation_edge_size,omitempty"`

// SearchEdgeSize represent the search edge size
SearchEdgeSize int `yaml:"search_edge_size" json:"search_edge_size"`
SearchEdgeSize int `yaml:"search_edge_size" json:"search_edge_size,omitempty"`

// AutoIndexDurationLimit represents auto indexing duration limit
AutoIndexDurationLimit string `yaml:"auto_index_duration_limit" json:"auto_index_duration_limit"`
AutoIndexDurationLimit string `yaml:"auto_index_duration_limit" json:"auto_index_duration_limit,omitempty"`

// AutoIndexCheckDuration represent checking loop duration about auto indexing execution
AutoIndexCheckDuration string `yaml:"auto_index_check_duration" json:"auto_index_check_duration"`
AutoIndexCheckDuration string `yaml:"auto_index_check_duration" json:"auto_index_check_duration,omitempty"`

// AutoSaveIndexDuration represent checking loop duration about auto save index execution
AutoSaveIndexDuration string `yaml:"auto_save_index_duration" json:"auto_save_index_duration"`
AutoSaveIndexDuration string `yaml:"auto_save_index_duration" json:"auto_save_index_duration,omitempty"`

// AutoIndexLength represent auto index length limit
AutoIndexLength int `yaml:"auto_index_length" json:"auto_index_length"`
AutoIndexLength int `yaml:"auto_index_length" json:"auto_index_length,omitempty"`

// InitialDelayMaxDuration represent maximum duration for initial delay
InitialDelayMaxDuration string `yaml:"initial_delay_max_duration" json:"initial_delay_max_duration"`
InitialDelayMaxDuration string `yaml:"initial_delay_max_duration" json:"initial_delay_max_duration,omitempty"`

// EnableInMemoryMode enables on memory ngt indexing mode
EnableInMemoryMode bool `yaml:"enable_in_memory_mode" json:"enable_in_memory_mode"`
EnableInMemoryMode bool `yaml:"enable_in_memory_mode" json:"enable_in_memory_mode,omitempty"`

// DefaultPoolSize represent default create index batch pool size
DefaultPoolSize uint32 `yaml:"default_pool_size" json:"default_pool_size"`
DefaultPoolSize uint32 `yaml:"default_pool_size" json:"default_pool_size,omitempty"`

// DefaultRadius represent default radius used for search
DefaultRadius float32 `yaml:"default_radius" json:"default_radius"`
DefaultRadius float32 `yaml:"default_radius" json:"default_radius,omitempty"`

// DefaultEpsilon represent default epsilon used for search
DefaultEpsilon float32 `yaml:"default_epsilon" json:"default_epsilon"`
DefaultEpsilon float32 `yaml:"default_epsilon" json:"default_epsilon,omitempty"`

// MinLoadIndexTimeout represents minimum duration of load index timeout
MinLoadIndexTimeout string `yaml:"min_load_index_timeout" json:"min_load_index_timeout"`
MinLoadIndexTimeout string `yaml:"min_load_index_timeout" json:"min_load_index_timeout,omitempty"`

// MaxLoadIndexTimeout represents maximum duration of load index timeout
MaxLoadIndexTimeout string `yaml:"max_load_index_timeout" json:"max_load_index_timeout"`
MaxLoadIndexTimeout string `yaml:"max_load_index_timeout" json:"max_load_index_timeout,omitempty"`

// LoadIndexTimeoutFactor represents a factor of load index timeout
LoadIndexTimeoutFactor string `yaml:"load_index_timeout_factor" json:"load_index_timeout_factor"`
LoadIndexTimeoutFactor string `yaml:"load_index_timeout_factor" json:"load_index_timeout_factor,omitempty"`

// EnableProactiveGC enables more proactive GC call for reducing heap memory allocation
EnableProactiveGC bool `yaml:"enable_proactive_gc" json:"enable_proactive_gc"`
EnableProactiveGC bool `yaml:"enable_proactive_gc" json:"enable_proactive_gc,omitempty"`

// VQueue represent the ngt vector queue buffer size
VQueue *VQueue `json:"vqueue,omitempty" yaml:"vqueue"`
}

// VQueue represent the ngt vector queue buffer size
type VQueue struct {
// InsertBufferSize represents insert channel buffer size
InsertBufferSize int `json:"insert_buffer_size,omitempty" yaml:"insert_buffer_size"`

// InsertBufferPoolSize represents insert time ordered slice buffer size
InsertBufferPoolSize int `json:"insert_buffer_pool_size,omitempty" yaml:"insert_buffer_pool_size"`

// DeleteBufferSize represents delete channel buffer size
DeleteBufferSize int `json:"delete_buffer_size,omitempty" yaml:"delete_buffer_size"`

// DeleteBufferPoolSize represents delete time ordered slice buffer size
DeleteBufferPoolSize int `json:"delete_buffer_pool_size,omitempty" yaml:"delete_buffer_pool_size"`
}

// Bind returns NGT object whose some string value is filed value or environment value.
Expand All @@ -92,5 +110,8 @@ func (n *NGT) Bind() *NGT {
n.MinLoadIndexTimeout = GetActualValue(n.MinLoadIndexTimeout)
n.MaxLoadIndexTimeout = GetActualValue(n.MaxLoadIndexTimeout)
n.LoadIndexTimeoutFactor = GetActualValue(n.LoadIndexTimeoutFactor)
if n.VQueue == nil {
n.VQueue = new(VQueue)
}
return n
}
9 changes: 8 additions & 1 deletion internal/config/ngt_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ func TestNGT_Bind(t *testing.T) {
AutoIndexLength int
InitialDelayMaxDuration string
EnableInMemoryMode bool
VQueue *VQueue
}
type want struct {
want *NGT
Expand Down Expand Up @@ -75,6 +76,7 @@ func TestNGT_Bind(t *testing.T) {
AutoIndexLength: 100,
InitialDelayMaxDuration: "1h",
EnableInMemoryMode: false,
VQueue: new(VQueue),
},
want: want{
want: &NGT{
Expand All @@ -91,6 +93,7 @@ func TestNGT_Bind(t *testing.T) {
AutoIndexLength: 100,
InitialDelayMaxDuration: "1h",
EnableInMemoryMode: false,
VQueue: new(VQueue),
},
},
},
Expand All @@ -110,6 +113,7 @@ func TestNGT_Bind(t *testing.T) {
AutoIndexLength: 100,
InitialDelayMaxDuration: "_initialDelayMaxDuration_",
EnableInMemoryMode: false,
VQueue: new(VQueue),
},
beforeFunc: func() {
_ = os.Setenv("indexPath", "config/ngt")
Expand Down Expand Up @@ -144,13 +148,16 @@ func TestNGT_Bind(t *testing.T) {
AutoIndexLength: 100,
InitialDelayMaxDuration: "1h",
EnableInMemoryMode: false,
VQueue: new(VQueue),
},
},
},
{
name: "returns NGT when all fields are empty",
want: want{
want: new(NGT),
want: &NGT{
VQueue: new(VQueue),
},
},
},
}
Expand Down
Loading