Series creation improvements

[jwilder]

This fixes a few performance issues I was seeing with series creation.

• There was lock contention on the series file when LogFile.execSeriesEntry was run. We already had the parsed series data further up the stack and LogFile.execSeriesEntry would need to look it up again and re-parse it. I tacked on the parsed series data to the LogEntry and execSeriesEntry will use it if exists, otherwise fall back to looking it up in the SeriesFile.
• Inserting to RHH was creating lots of little allocations. I switched it to use some temp buffers that can be re-used.
• The mmap.Map call madvise(MADV_RANDOM) which was trigger a lot of read IOPS on the series files. I removed that and seems to reduce read IOPS.
• Loading a big TSM file was also trigger lots of read IOPS due to page faults when scanning the index. I added a hint madvise(MADV_WILLNEED) that helps a bit. I think a better solution would be to store some extra info in the TSM index to avoid scanning at load time.

Required for all non-trivial PRs

• Rebased/mergable
• Tests pass
• CHANGELOG.md updated
• Sign CLA (if not already signed)

[e-dard]

LGTM 👍

I have some comments on the mmap stuff but not sure if it's relevant or not.

[e-dard]

We mmap in (I think) five places in the database:

1. We mmap TSI index files (.tsi).
2. We mmap TSI log files (.tsl).
3. We mmap the Series File hashmap index (_series/*/index).
4. We mmap the Series File segment files (_series/*/0000).
5. We mmap the TSM index (.tsm).

As I understand it, SYS_MADVISE is there to inform the kernel about how the memory within the mmap'd data will be accessed. The current setting is telling the kernel that we will need to randomly access pages, which are typically going to be 4KB of the mapped data.

For the above five examples I think that the way we access pages would probably be different.

1. We probably sequentially scan a lot when pulling series ids off, but then we probably jump around some parts because we have things like hash indexes and sketches and so on mapped in.
2. Unless I'm mistaken we're only going to sequentially read this file, to process the log entries.
3. I would expect this file to be basically randomly accessed. While there would be might be some sequential access, e.g., when probing, wouldn't it likely all be within a single page or at most a few?
4. These files are probably mainly randomly accessed via the offsets of the series id in the segments.
5. I'm not familiar enough with the tsm index to know its access patterns.

To me, it seems like we should be passing in the advice values to Map depending on the expected use-case. It looks like we might have a mixture of MADV_NORMAL, MADV_SEQUENTIAL and MADV_RANDOM.

@jwilder I guess when you switched from MADV_RANDOM to MADV_NORMAL then the kernel was doing some read ahead that was reducing the number of IO read operations. Though, of course, if we used MADV_SEQUENTIAL then more aggressive read ahead would be done and read IOPS might be reduced even further.

I think we could experiment with different advise parameters for the different subsystems that use mmap and set the advise value accordingly.

[jwilder]

I've had to remove any place where we're using MADV_RANDOM in the past due to adverse performance problems. See: #8872 and #5221. Other projects (boltdb/bolt#691) have seen similar issues.

I only add madavise calls in specific spots after testing them quite extensively and can see that they improve what I'm measuring. Despite what the docs say they do, it's kind of a black box as to what kernel actually does.

[e-dard]

@jwilder I guess what I'm saying is: we should do some explicit experimentation in the future on all the sub systems and see if any advice other than NORMAL is appropriate anywhere. If we're doing really sequential stuff then MADV_SEQUENTIAL might be even better for read IOPS than MADV_NORMAL for example.

[e-dard]

Could make this function a little shorter like this?

func assign(x, v []byte) []byte {
	if cap(x) < len(v) {
		x = make([]byte, len(v))
	}
	copy(x, v)
	return x[:len(v)]
}

[e-dard]

Could we add this call to the Series File index? We will always need to read from that file too right?

[jwilder]

I was not measuring series file IOPS so I would not add this there without seeing the actual change in performance. This does have some benefit with larger TSM files that are not in the page cache though.