Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP [dbnode] Refactor posting list operations to be lazily evaluated and zero copy #2791

Open
wants to merge 119 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
f97ec5f
[dbnode] Reduce index query allocations
robskillington Oct 22, 2020
00462c1
Merge branch 'master' into r/reduce-index-query-allocs
robskillington Oct 22, 2020
17b7398
Use new LRU cache and instrument
robskillington Oct 23, 2020
a6f05a7
Revert go.mod
robskillington Oct 23, 2020
c8683af
Use read only bitmap by default in FST segment
robskillington Oct 25, 2020
a0f1801
Implement read only bitmap for close to zero alloc operations on post…
robskillington Oct 26, 2020
5208869
Add tests
robskillington Oct 27, 2020
a76c6c4
Fix tests
robskillington Oct 28, 2020
9592410
Fix few issues
robskillington Oct 28, 2020
a0ce92f
Fix multi segments builder
robskillington Oct 28, 2020
6786415
Fix fieldsAndTermsIter
robskillington Oct 28, 2020
dd58315
Fix checker
robskillington Oct 28, 2020
cfac389
Merge branch 'master' into r/reduce-index-query-allocs
arnikola Oct 30, 2020
fd8e4e5
Fix multi-intersect searches that start with empty posting lists
robskillington Nov 1, 2020
f2688e0
Refactor read only bitmap range to own datastructure
robskillington Nov 2, 2020
c1fbb80
Add ability for RO postings lists to be enabled by env var
robskillington Nov 3, 2020
e1b5360
Merge branch 'master' into r/reduce-index-query-allocs
robskillington Nov 3, 2020
861fcdf
Fix integration test kickoff
robskillington Nov 3, 2020
6396865
Fix multi bitmap iterator and also implement field filtering for aggr…
robskillington Nov 3, 2020
8d74ee1
Fix field filtering based on restrict by tags query
robskillington Nov 3, 2020
8da22f2
Close iterator just once
robskillington Nov 3, 2020
4ea2ec6
Fix integration test
robskillington Nov 3, 2020
dbf3148
Add pooling of regexp searcher and multi bitmap iterators
robskillington Nov 4, 2020
339b4e7
Cache loaded vellum FST structs and reuse readers
robskillington Nov 4, 2020
91afefa
Add generated map types
robskillington Nov 4, 2020
8d989e2
Fix unlock
robskillington Nov 4, 2020
9160bd4
Merge branch 'master' into r/reduce-index-query-allocs
robskillington Nov 4, 2020
2108f7c
Merge branch 'master' into r/reduce-index-query-allocs
arnikola Nov 5, 2020
7a2683a
Merge branch 'master' into r/reduce-index-query-allocs
arnikola Nov 5, 2020
ee07598
Reuse containers used for mutable postings lists
robskillington Nov 9, 2020
0805169
Merge branch 'r/reduce-index-query-allocs' of github.com:m3db/m3 into…
robskillington Nov 9, 2020
d0cc13f
Apply capacity fix
robskillington Nov 9, 2020
fd2af0c
Add concurrency for queries with lost of segments and pool postings l…
robskillington Nov 10, 2020
099ecd7
Merge branch 'master' into r/reduce-index-query-allocs
robskillington Nov 10, 2020
d82a47d
Use separate pools for querying segments separately
robskillington Nov 10, 2020
e113d5b
Fix build
robskillington Nov 11, 2020
9d61cb2
Wire up new series index backoff to runtime options
robskillington Nov 12, 2020
dd38ead
[dbnode] Refactor wide query path (#2826)
arnikola Nov 11, 2020
a3bd18a
[dbnode] Introduce Aggregator type (#2840)
linasm Nov 11, 2020
3b5c0ff
[coordinator] Set default namespace tag to avoid colliding with commo…
robskillington Nov 12, 2020
24d79f8
Merge branch 'master' into r/reduce-index-query-allocs
robskillington Nov 12, 2020
567dd45
Add read through caching of searches to segments (including foreground)
robskillington Nov 14, 2020
3f46cfa
Merge branch 'r/reduce-index-query-allocs' of github.com:m3db/m3 into…
robskillington Nov 14, 2020
31d5816
Fix typo
robskillington Nov 14, 2020
7da3dd5
Set postings list pooling
robskillington Nov 15, 2020
1084aa8
Propagate postings list cache
robskillington Nov 16, 2020
03203b7
Use independent postings list cache for searches
robskillington Nov 16, 2020
36a1690
Use concurrent cache and striped context finalizers
robskillington Nov 18, 2020
0af1add
Use const for array size
robskillington Nov 18, 2020
a41e893
Remove test unsafe
robskillington Nov 18, 2020
ef5dd15
Rename to slot index
robskillington Nov 18, 2020
7f50c75
Add more realtime stats to number of mutable segments
robskillington Nov 18, 2020
8beb1a1
Reduce contention for add documents and query cache key building
robskillington Nov 19, 2020
6057969
Only acquire lock during stats update/read
robskillington Nov 19, 2020
4af3aba
Fix race with statsNoLock
robskillington Nov 19, 2020
178f02f
Always lock even if using non-concurrent builder
robskillington Nov 19, 2020
ccf9035
Force cold writes enabled
robskillington Dec 8, 2020
0483b50
Force cold writes enabled
robskillington Dec 8, 2020
2bbf6ba
Use fast IntersectsAny code path for aggregate queries
robskillington Dec 8, 2020
09eeb3b
Merge branch 'r/reduce-index-query-allocs' of github.com:m3db/m3 into…
robskillington Dec 8, 2020
4dbe593
Remove field not present for storage options
robskillington Dec 8, 2020
4cb4029
Fix non readonly postings
robskillington Dec 9, 2020
0b4db71
Fix non readonly postings
robskillington Dec 9, 2020
f668a84
Fix intersect array
robskillington Dec 9, 2020
a8eaf9a
Add in-memory block index block to replace per time window block, pha…
robskillington Jan 14, 2021
78eaa8b
Create the in memory block earlier
robskillington Jan 16, 2021
5f93d0c
Avoid indexing already indexed timeseries
robskillington Jan 20, 2021
7ce03c0
Add instrumentation
robskillington Jan 20, 2021
416dbfc
Fix build
robskillington Jan 20, 2021
71d54ca
Always query the in-memory block
robskillington Jan 20, 2021
95ca25f
Only run a compaction on a background segment if active block starts …
robskillington Jan 22, 2021
9fd2d4e
Don't require acquiring other locks from maybeBackgroundCompactWithLock
robskillington Jan 22, 2021
e98f77e
Add parallel background compaction
robskillington Jan 23, 2021
9857326
Add wg.done()
robskillington Jan 23, 2021
23e5ff2
Allocate number of background compactors
robskillington Jan 23, 2021
23f6548
Use up to num cpu / 2 for background compaction threads
robskillington Jan 23, 2021
1fb1850
Use num CPU minus one for background compactors
robskillington Jan 23, 2021
a333625
Cast num CPU minus one to int
robskillington Jan 23, 2021
ecbd20e
Use default compaction level of 0-500k
robskillington Jan 23, 2021
fe9eab3
Sort largest to smallest segments when using multi segments builder
robskillington Jan 23, 2021
d41284d
Use maps solely for determining existence for compaction purposes or not
robskillington Jan 24, 2021
e969766
Use indexedBloomFilterByTimeLock lock
robskillington Jan 24, 2021
c25a7bc
Fix cast to interface
robskillington Jan 24, 2021
976bf85
Allocation indexedSnapshot at creation
robskillington Jan 24, 2021
6284b7b
Fix build
robskillington Jan 24, 2021
0aed2f5
Allocate backgroundCompactIndexedSnapshot when creating mutable segments
robskillington Jan 25, 2021
b961dea
Remove lock acquiring to check indexed snapshot
robskillington Jan 25, 2021
16a1c54
Call NotifySealedBlocks out of lock and only do the full check of whe…
robskillington Jan 25, 2021
193c4f3
Separate GCing background segments from foreground segments
robskillington Jan 25, 2021
cdd8795
Use separate number of compactors per type of background compaction
robskillington Jan 25, 2021
5eb19b6
Alloc GC segments each background GC segment run, also do not mutate …
robskillington Jan 27, 2021
ccb3035
Reduce insert lock contention by spreading over many queues per CPU
robskillington Jan 29, 2021
de3ce9a
4x the queues per CPU core
robskillington Jan 29, 2021
834efa6
Use series lookup as source of truth for if indexed or not and if sho…
robskillington Feb 7, 2021
3699014
Only run background GC when sealed blocks updated
robskillington Feb 8, 2021
4934faf
Remove purge expired series warning
robskillington Feb 8, 2021
55632a9
Use o(1) lookup for negative offsets for postings IDs in multisegment…
robskillington Feb 9, 2021
3b8c1d0
Load term FSTs up front to avoid slow queries during compactions
robskillington Feb 10, 2021
1c86639
Fix CI tests
robskillington Feb 26, 2021
33590cc
Always consider the in-memory block
robskillington Feb 26, 2021
6a0aa10
Filter instead of shrink slice
robskillington Mar 8, 2021
9421bae
Only decrement readerwrite count if marking as success
robskillington Mar 10, 2021
daed491
Always re-resolve entry when checking if should GC series
robskillington Mar 12, 2021
3bc3b81
Merge commit 'e0bc12aeda246f2c39852d44ddb59a11cb3d9d31' into r/reduce…
robskillington Mar 29, 2021
3de1690
Avoid index block ticking while holding index write lock
robskillington Apr 8, 2021
da1aeb4
Rerun cached searches when replacing background segments to avoid nee…
robskillington Apr 6, 2021
0738d68
Add concurrent execution of cached searches
robskillington Apr 6, 2021
1f8a510
Initialize worker pool
robskillington Apr 6, 2021
9cf50ee
Reuse cached postings when populating cached searches
robskillington Apr 10, 2021
e151cbc
Fix search query proto
robskillington Apr 10, 2021
b886052
Fix missing postings for registry value
robskillington Apr 10, 2021
a867722
Add a postings cache TTL of 15min, also sort conjunction query for de…
robskillington Apr 12, 2021
b8f4d82
Fix build
robskillington Apr 12, 2021
34a8594
Restore simple LRU cache for accuracy when compacting segments
robskillington Apr 13, 2021
61e6185
Shard postings list cache
robskillington Apr 13, 2021
5e03570
Fix reconstructed searches not being executed
robskillington Apr 15, 2021
734d67f
Do not pool multi-bitmap iterators
robskillington Apr 16, 2021
43c0026
Add test for populate worker
robskillington Apr 16, 2021
6379d0b
If tag name is a graphite tag then do not allocate each time
robskillington May 12, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ steps:
run: app
workdir: /go/src/github.com/m3db/m3
<<: *common
- name: "Integration (:docker:)"
- name: "Integration (:docker:, postings=default) %n"
command: make clean install-vendor-m3 docker-integration-test
parallelism: 2
env:
Expand All @@ -61,6 +61,16 @@ steps:
gopath-checkout#v1.0.1:
import: github.com/m3db/m3
<<: *common
- name: "Integration (:docker:, postings=read_only) %n"
command: M3DB_READ_ONLY_POSTINGS=true make clean install-vendor-m3 docker-integration-test
parallelism: 2
env:
CGO_ENABLED: 0
GIMME_GO_VERSION: 1.13.x
plugins:
gopath-checkout#v1.0.1:
import: github.com/m3db/m3
<<: *common
- name: "Prometheus compatibility (:docker:)"
command: make clean install-vendor-m3 docker-compatibility-test
parallelism: 1
Expand All @@ -71,17 +81,17 @@ steps:
gopath-checkout#v1.0.1:
import: github.com/m3db/m3
<<: *common
- name: "Integration (dbnode Recently Read) %n"
- name: "Integration (dbnode cache=LRU postings=default) %n"
parallelism: 2
command: make clean install-vendor-m3 test-ci-integration-dbnode cache_policy=recently_read
command: make clean install-vendor-m3 test-ci-integration-dbnode cache_policy=lru m3db_postings_ro=false
plugins:
docker-compose#v2.5.1:
run: app
workdir: /go/src/github.com/m3db/m3
<<: *common
- name: "Integration (dbnode LRU) %n"
- name: "Integration (dbnode cache=LRU postings=read_only) %n"
parallelism: 2
command: make clean install-vendor-m3 test-ci-integration-dbnode cache_policy=lru
command: make clean install-vendor-m3 test-ci-integration-dbnode cache_policy=lru m3db_postings_ro=true
plugins:
docker-compose#v2.5.1:
run: app
Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ thrift_rules_dir := generated/thrift
vendor_prefix := vendor
cache_policy ?= recently_read
genny_target ?= genny-all
m3db_postings_ro ?= false # Remove after MigrationReadOnlyPostings done

BUILD := $(abspath ./bin)
VENDOR := $(m3_package_path)/$(vendor_prefix)
Expand Down Expand Up @@ -256,8 +257,7 @@ SUBDIR_TARGETS := \
asset-gen \
genny-gen \
license-gen \
all-gen \
all-gen
all-gen

.PHONY: test-ci-unit
test-ci-unit: test-base
Expand Down Expand Up @@ -359,7 +359,7 @@ test-ci-big-unit-$(SUBDIR):
.PHONY: test-ci-integration-$(SUBDIR)
test-ci-integration-$(SUBDIR):
@echo "--- test-ci-integration $(SUBDIR)"
SRC_ROOT=./src/$(SUBDIR) PANIC_ON_INVARIANT_VIOLATED=true INTEGRATION_TIMEOUT=10m TEST_SERIES_CACHE_POLICY=$(cache_policy) make test-base-ci-integration
SRC_ROOT=./src/$(SUBDIR) PANIC_ON_INVARIANT_VIOLATED=true INTEGRATION_TIMEOUT=10m TEST_SERIES_CACHE_POLICY=$(cache_policy) M3DB_READ_ONLY_POSTINGS=$(m3db_postings_ro) make test-base-ci-integration
@echo "--- uploading coverage report"
$(codecov_push) -f $(coverfile) -F $(SUBDIR)

Expand Down
3 changes: 2 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ require (
github.com/cheekybits/genny v1.0.0 // indirect
github.com/containerd/continuity v0.0.0-20200413184840-d3ef23f19fbb // indirect
github.com/davecgh/go-spew v1.1.1
github.com/dgraph-io/ristretto v0.0.3
github.com/fortytw2/leaktest v1.2.1-0.20180901000122-b433bbd6d743
github.com/fossas/fossa-cli v1.0.30
github.com/garethr/kubeval v0.0.0-20180821130434-c44f5193dc94
Expand Down Expand Up @@ -55,7 +56,7 @@ require (
github.com/m3db/stackadler32 v0.0.0-20180104200216-bfebcd73ef6f
github.com/m3db/stackmurmur3/v2 v2.0.2
github.com/m3db/tools v0.0.0-20181008195521-c6ded3f34878
github.com/m3dbx/pilosa v1.4.1
github.com/m3dbx/pilosa v1.4.2-0.20201109081833-6c9df43642fd
github.com/m3dbx/vellum v0.0.0-20201119082309-5b47f7a70f69
github.com/mauricelam/genny v0.0.0-20180903214747-eb2c5232c885
github.com/mjibson/esc v0.1.0
Expand Down
8 changes: 6 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,11 @@ github.com/daviddengcn/go-villa v0.0.0-20160111144444-3f35da8ba982 h1:2Trx4ntMtx
github.com/daviddengcn/go-villa v0.0.0-20160111144444-3f35da8ba982/go.mod h1:U8xNoHcXfPnZzy9zCxeKRjaJgC1d3613rFHjZVVAqKc=
github.com/denis-tingajkin/go-header v0.3.1 h1:ymEpSiFjeItCy1FOP+x0M2KdCELdEAHUsNa8F+hHc6w=
github.com/denis-tingajkin/go-header v0.3.1/go.mod h1:sq/2IxMhaZX+RRcgHfCRx/m0M5na0fBt4/CRe7Lrji0=
github.com/dgraph-io/ristretto v0.0.3 h1:jh22xisGBjrEVnRZ1DVTpBVQm0Xndu8sMl0CWDzSIBI=
github.com/dgraph-io/ristretto v0.0.3/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E=
github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/dgryski/go-sip13 v0.0.0-20190329191031-25c5027a8c7b/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ=
Expand Down Expand Up @@ -650,8 +653,8 @@ github.com/m3db/thrift v0.0.0-20190820191926-05b5a2227fe4 h1:1x3mMuURd3wqKJ2qVjh
github.com/m3db/thrift v0.0.0-20190820191926-05b5a2227fe4/go.mod h1:xVfRinGzD3cYDRvMjy6RkIwM+iNL2KHNLZjT0VpVZT8=
github.com/m3db/tools v0.0.0-20181008195521-c6ded3f34878 h1:kww0LtVVfGrXR7Ofpbi/9bvc2EGYMQC0LCH/gQXoolE=
github.com/m3db/tools v0.0.0-20181008195521-c6ded3f34878/go.mod h1:TxroQUZzb1wzOsq+4+TfVtT7z89YTz3v2UJAYfLNfLE=
github.com/m3dbx/pilosa v1.4.1 h1:/Cpp1XAHSd6orpjceXGiKpCoDdYBP5BD/6NoqGG9eVg=
github.com/m3dbx/pilosa v1.4.1/go.mod h1:Jt0+w9O08sa7qWDeRC58VBjb4OeOTDMOhfvVmyeVCO8=
github.com/m3dbx/pilosa v1.4.2-0.20201109081833-6c9df43642fd h1:C+RCSMuplTpLH8Fiwb87XIkbYEabVf9itroR9+u4RYo=
github.com/m3dbx/pilosa v1.4.2-0.20201109081833-6c9df43642fd/go.mod h1:Jt0+w9O08sa7qWDeRC58VBjb4OeOTDMOhfvVmyeVCO8=
github.com/m3dbx/vellum v0.0.0-20201119082309-5b47f7a70f69 h1:dANuca0xuYlZR7qWdPIIAZKG0YHvsbLTzdenj53yQvc=
github.com/m3dbx/vellum v0.0.0-20201119082309-5b47f7a70f69/go.mod h1:DOTAUfV4bzK6Nrb0dboT/oCG0DnQuX+/n0jfZPh6xxI=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
Expand Down Expand Up @@ -1197,6 +1200,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e h1:vcxGaoTs7kV8m5Np9uUNQin4
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208 h1:qwRHBd0NqMbJxfbotnDhm2ByMI1Shq4Y6oRJo21SGJA=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
golang.org/x/sys v0.0.0-20200826173525-f9321e4c35a6 h1:DvY3Zkh7KabQE/kfzMvYvKirSiguP9Q/veMtkYyf0o8=
golang.org/x/sys v0.0.0-20200826173525-f9321e4c35a6/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
m3coordinator01:
expose:
- "7202"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
m3coordinator01:
expose:
- "7202"
Expand Down
2 changes: 2 additions & 0 deletions scripts/docker-integration-tests/carbon/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator01:
expose:
- "7201"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator01:
expose:
- "7201"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator01:
expose:
- "7201"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=dbnode01
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode.yml:/etc/m3dbnode/m3dbnode.yml"
etcd01:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_a_m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-a.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_a_dbnode02:
Expand All @@ -26,6 +27,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_a_m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-a.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_a_coordinator01:
Expand Down Expand Up @@ -54,6 +56,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_b_m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-b.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_b_dbnode02:
Expand All @@ -68,6 +71,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_b_m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-b.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_b_coordinator01:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator01:
expose:
- "7201"
Expand Down
22 changes: 20 additions & 2 deletions scripts/docker-integration-tests/prometheus/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -348,8 +348,10 @@ function test_query_restrict_tags {

# First write some hidden metrics.
echo "Test write with unaggregated metrics type works as expected"
TAG_NAME_0="restricted_metrics_type" TAG_VALUE_0="hidden" \
TAG_NAME_1="foo_tag" TAG_VALUE_1="foo_tag_value" \
TAG_NAME_0="__name__" TAG_VALUE_0="hidden_metric_by_default" \
TAG_NAME_1="restricted_metrics_type" TAG_VALUE_1="hidden" \
TAG_NAME_2="hidden_label_by_default" TAG_VALUE_2="hidden" \
TAG_NAME_3="foo_tag" TAG_VALUE_3="foo_tag_value" \
prometheus_remote_write \
some_hidden_metric now 42.42 \
true "Expected request to succeed" \
Expand All @@ -366,6 +368,22 @@ function test_query_restrict_tags {
echo "Test restrict by tags with coordinator defaults"
ATTEMPTS=5 TIMEOUT=2 MAX_TIMEOUT=4 retry_with_backoff \
'[[ $(curl -s 0.0.0.0:7201/api/v1/query?query=\\{restricted_metrics_type=\"hidden\"\\} | jq -r ".data.result | length") -eq 0 ]]'

# Check we can see metric in label values endpoint with zero restrictions applied.
ATTEMPTS=5 TIMEOUT=2 MAX_TIMEOUT=4 retry_with_backoff \
'[[ $(curl -s -H "M3-Restrict-By-Tags-JSON: {}" 0.0.0.0:7201/api/v1/label/__name__/values | jq ".data | map(select(. == \"hidden_metric_by_default\")) | length") -eq 1 ]]'

# Now check that without easing restrictions that we can see the label value in question.
ATTEMPTS=5 TIMEOUT=2 MAX_TIMEOUT=4 retry_with_backoff \
'[[ $(curl -s 0.0.0.0:7201/api/v1/label/__name__/values | jq ".data | map(select(. == \"hidden_metric_by_default\")) | length") -eq 0 ]]'

# Check we can see metric in labels endpoint with zero restrictions applied.
ATTEMPTS=5 TIMEOUT=2 MAX_TIMEOUT=4 retry_with_backoff \
'[[ $(curl -s -H "M3-Restrict-By-Tags-JSON: {}" 0.0.0.0:7201/api/v1/labels | jq ".data | map(select(. == \"hidden_label_by_default\")) | length") -eq 1 ]]'

# Now check that without easing restrictions that we can see the label in question.
ATTEMPTS=5 TIMEOUT=2 MAX_TIMEOUT=4 retry_with_backoff \
'[[ $(curl -s 0.0.0.0:7201/api/v1/labels | jq ".data | map(select(. == \"hidden_label_by_default\")) | length") -eq 0 ]]'
}

function test_series {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator01:
expose:
- "7201"
Expand All @@ -32,6 +34,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator02:
expose:
- "17201"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator-cluster-a:
expose:
- "7201"
Expand All @@ -34,6 +36,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator-cluster-b:
expose:
- "7201"
Expand All @@ -58,6 +62,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator-cluster-c:
expose:
- "7201"
Expand Down
2 changes: 2 additions & 0 deletions scripts/docker-integration-tests/repair/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode.yml:/etc/m3dbnode/m3dbnode.yml"
dbnode02:
Expand All @@ -26,6 +27,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode.yml:/etc/m3dbnode/m3dbnode.yml"
coordinator01:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_a_m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-a.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_a_dbnode02:
Expand All @@ -26,6 +27,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_a_m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-a.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_a_coordinator01:
Expand Down Expand Up @@ -54,6 +56,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_b_m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-b.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_b_dbnode02:
Expand All @@ -68,6 +71,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_b_m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-b.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_b_coordinator01:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_a_m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-a.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_a_dbnode02:
Expand All @@ -26,6 +27,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_a_m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-a.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_a_coordinator01:
Expand Down Expand Up @@ -54,6 +56,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_b_m3db_local_1
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-b.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_b_dbnode02:
Expand All @@ -68,6 +71,7 @@ services:
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_HOST_ID=cluster_b_m3db_local_2
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
volumes:
- "./m3dbnode-cluster-b.yml:/etc/m3dbnode/m3dbnode.yml"
cluster_b_coordinator01:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ services:
networks:
- backend
image: "m3dbnode_integration:${REVISION}"
environment:
- M3DB_READ_ONLY_POSTINGS # Remove after MigrationReadOnlyPostings done
coordinator01:
expose:
- "7201"
Expand Down
3 changes: 2 additions & 1 deletion src/aggregator/client/m3msg_client_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
package client

import (
"testing"
"os"
"strings"

"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
Expand Down
Loading