Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reload search blocks and replay search WAL #1000

Merged
merged 23 commits into from
Oct 6, 2021
Merged

Reload search blocks and replay search WAL #1000

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
cdec27a
Checkpoint: Initial implementation of v2 search WAL
annanay25 Sep 29, 2021
6738a72
better handling of reload of blocks without search data
annanay25 Sep 29, 2021
52e5f61
Checkpoint
annanay25 Sep 30, 2021
4e05804
another commit another wal replay
annanay25 Sep 30, 2021
dfbbd02
wip: ingester search test
annanay25 Sep 30, 2021
5fa8cc9
Fix Rescan search blocks, move ParseFilename into wal folder
annanay25 Oct 1, 2021
1823cb1
Append block uses new ParseFilename
annanay25 Oct 1, 2021
18d2e64
Add tests, benchmarks, pass encoding along correctly
annanay25 Oct 1, 2021
481d51f
Changelog
annanay25 Oct 1, 2021
82cab7f
Merge branch 'main' into reload-search-blocks
annanay25 Oct 1, 2021
99506a3
Post merge cleanup
annanay25 Oct 1, 2021
98c7e1b
Err handling for search disabled
annanay25 Oct 1, 2021
29d8eaf
Use the right level package, reload backend search blocks
annanay25 Oct 1, 2021
195b2fa
never refactor variables using an ide
annanay25 Oct 1, 2021
0747e09
Address comments, fix test
annanay25 Oct 1, 2021
b5e5941
Reuse StreamingSearchBlock iterator in search, relocate dedupe test
mdisibio Oct 4, 2021
5a163fe
Make wal search encoding configurable, default to gzip like backend b…
mdisibio Oct 5, 2021
de3a1e2
Make wal search encoding configurable, default to gzip like backend b…
mdisibio Oct 5, 2021
1aaca42
Simplify some search tests which were doing more work than seemed nec…
mdisibio Oct 5, 2021
5b7485f
Comment out flaky test as discussed
mdisibio Oct 5, 2021
97dafa2
Code review suggestions
mdisibio Oct 6, 2021
7b6a81f
Code review suggestions, add tests for ParseFileName
mdisibio Oct 6, 2021
7d6a3b2
Code review suggestions
mdisibio Oct 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion tempodb/wal/wal.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,8 @@ func (w *WAL) NewFile(blockid uuid.UUID, tenantid string, dir string) (*os.File,
return os.OpenFile(filepath.Join(p, filename), os.O_CREATE|os.O_RDWR, 0644)
}

// ParseFilename returns (blockID, tenant, version, encoding, dataEncoding, error)
// ParseFilename returns (blockID, tenant, version, encoding, dataEncoding, error).
// Example: "00000000-0000-0000-0000-000000000000:1:v2:snappy:v1"
func ParseFilename(filename string) (uuid.UUID, string, string, backend.Encoding, string, error) {
splits := strings.Split(filename, ":")

Expand Down
48 changes: 48 additions & 0 deletions tempodb/wal/wal_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package wal
import (
"bytes"
"context"
"errors"
"io"
"math/rand"
"os"
Expand Down Expand Up @@ -269,6 +270,53 @@ func testAppendReplayFind(t *testing.T, e backend.Encoding) {
require.NoError(t, err)
}

func TestParseFileName(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a test for ParseFileName in append_block_test.go. But no worries, we can club the two in a later PR


testCases := []struct {
name string
filename string
blockID uuid.UUID
tenant string
version string
encoding backend.Encoding
dataEncoding string
err error
}{
{
"wal",
"00000000-0000-0000-0000-000000000000:1:v2:snappy:v1",
uuid.MustParse("00000000-0000-0000-0000-000000000000"), "1", "v2", backend.EncSnappy, "v1", nil,
},
{
"search wal",
"00000000-0000-0000-0000-000000000000:1:v2:snappy:",
uuid.MustParse("00000000-0000-0000-0000-000000000000"), "1", "v2", backend.EncSnappy, "", nil,
},
{
"missing segments",
"xyz",
uuid.Nil, "", "", backend.EncNone, "", errors.New("unable to parse xyz. unexpected number of segments"),
},
{
"no data encoding",
"00000000-0000-0000-0000-000000000000:1:v2:snappy",
uuid.MustParse("00000000-0000-0000-0000-000000000000"), "1", "v2", backend.EncSnappy, "", nil,
},
}

for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
b, tn, v, e, de, err := ParseFilename(tc.filename)
require.Equal(t, tc.blockID, b)
require.Equal(t, tc.tenant, tn)
require.Equal(t, tc.version, v)
require.Equal(t, tc.encoding, e)
require.Equal(t, tc.dataEncoding, de)
require.Equal(t, tc.err, err)
})
}
}

func BenchmarkWALNone(b *testing.B) {
benchmarkWriteFindReplay(b, backend.EncNone)
}
Expand Down