Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dbnode] Add DocRef() to NS and multiple series metadata types #2931

Merged
merged 3 commits into from
Nov 21, 2020
Merged

[dbnode] Add DocRef() to NS and multiple series metadata types #2931

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
535f63c
Add docref to db ns.
notbdu Nov 20, 2020
b749b66
Extend the on flush new series event to account for differing types o…
notbdu Nov 20, 2020
2fd2d35
Address nits.
notbdu Nov 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions src/dbnode/persist/fs/merger.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,10 +249,15 @@ func (m *merger) Merge(

if err == nil {
err = onFlush.OnFlushNewSeries(persist.OnFlushNewSeriesEvent{
Shard: shard,
BlockStart: startTime,
FirstWrite: mergeWithData.FirstWrite,
SeriesMetadata: seriesMetadata,
Shard: shard,
BlockStart: startTime,
FirstWrite: mergeWithData.FirstWrite,
SeriesMetadata: persist.SeriesMetadata{
Type: persist.SeriesDocumentType,
Document: seriesMetadata,
// The lifetime of the shard series metadata is longly lived.
LifeTime: persist.SeriesMetadataLifeTimeLong,
},
})
}

Expand Down
43 changes: 39 additions & 4 deletions src/dbnode/persist/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,7 @@ import (
"github.com/pborman/uuid"
)

var (
errReuseableTagIteratorRequired = errors.New("reuseable tags iterator is required")
)
var errReuseableTagIteratorRequired = errors.New("reuseable tags iterator is required")

// Metadata is metadata for a time series, it can
// have several underlying sources.
Expand Down Expand Up @@ -322,12 +320,49 @@ const (
FileSetIndexContentType
)

// SeriesMetadataLifeTime describes the memory life time type.
type SeriesMetadataLifeTime uint8

const (
// SeriesMetadataLifeTimeLong means the underlying memory's life time is long lived and exceeds
// the execution duration of the series metadata receiver.
SeriesMetadataLifeTimeLong SeriesMetadataLifeTime = iota
// SeriesMetadataLifeTimeShortLived means that the underlying memory is only valid for the duration
// of the OnFlushNewSeries call. Must clone the underlying bytes in order to extend the life time.
SeriesMetadataLifeTimeShortLived
)

// SeriesMetadataType describes the type of series metadata.
type SeriesMetadataType uint8

const (
// SeriesDocumentType means the metadata is in doc.Document form.
SeriesDocumentType SeriesMetadataType = iota
// SeriesIDAndEncodedTagsType means the metadata is in IDAndEncodedTags form.
SeriesIDAndEncodedTagsType
)

// IDAndEncodedTags contains a series ID and encoded tags.
type IDAndEncodedTags struct {
ID ident.BytesID
EncodedTags ts.EncodedTags
}

// SeriesMetadata captures different representations of series metadata and
// the ownership status of the underlying memory.
type SeriesMetadata struct {
Document doc.Document
IDAndEncodedTags IDAndEncodedTags
Type SeriesMetadataType
LifeTime SeriesMetadataLifeTime
}

// OnFlushNewSeriesEvent is the fields related to a flush of a new series.
type OnFlushNewSeriesEvent struct {
Shard uint32
BlockStart time.Time
FirstWrite time.Time
SeriesMetadata doc.Document
SeriesMetadata SeriesMetadata
}

// OnFlushSeries performs work on a per series level.
Expand Down
9 changes: 9 additions & 0 deletions src/dbnode/storage/namespace.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ import (
"github.com/m3db/m3/src/dbnode/ts"
"github.com/m3db/m3/src/dbnode/ts/writes"
"github.com/m3db/m3/src/dbnode/x/xio"
"github.com/m3db/m3/src/m3ninx/doc"
"github.com/m3db/m3/src/x/clock"
"github.com/m3db/m3/src/x/context"
xerrors "github.com/m3db/m3/src/x/errors"
Expand Down Expand Up @@ -1793,3 +1794,11 @@ func (n *dbNamespace) aggregateTiles(

return processedTileCount, nil
}

func (n *dbNamespace) DocRef(id ident.ID) (doc.Document, bool, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've been sticking to the type BytesID []byte type alias everywhere in this whole flow so far, successfully.
Do I understand it correctly that passing it as an interface will cause allocation? Sadly, this might be difficult to avoid here, but still, something that I want to bring up.

Copy link
Collaborator

@robskillington robskillington Nov 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's true, you can use a re-useable ID from the caller if you like though (not relevant to this PR, but how to use this PR):
https://godoc.org/github.com/m3db/m3/src/x/ident#NewReuseableBytesID

var bytesID ident.BytesID

id := ident.NewReuseableBytesID() // Reuse this on consequent calls
id.Reset(bytesID)

d, ok, err := ns.DocRef(id)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea it will. I'm going to be pulling and putting the from the identifier pool.

shard, _, err := n.readableShardFor(id)
if err != nil {
return doc.Document{}, false, err
}
return shard.DocRef(id)
}
32 changes: 32 additions & 0 deletions src/dbnode/storage/storage_mock.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions src/dbnode/storage/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,9 @@ type Namespace interface {

// SetReadOnly sets the value of ReadOnly option.
SetReadOnly(value bool)

// DocRef returns the doc if already present in a namespace shard.
DocRef(id ident.ID) (doc.Document, bool, error)
}

// NamespacesByID is a sortable slice of namespaces by ID.
Expand Down