eventpb: make the Timestamp field an int64 #58070
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Release note: None
|
This change is |
knz
force-pushed
the
20201218-numeric-timestamps
branch
from
d22fcb9
to
40c183f
Compare
While exploring how to use the data in practice, I noticed that the timestamp was encoded as a string in the JSON output, and that was making the events hard to parse and to use. There are many more functions that can operate on a number. Moreover, the integer representation is just more compact. Release note (api change): The 'Timestamp' field of structured notable events is now numeric, and encodes a number of nanoseconds since the unix epoch. (Note that this API has not yet been published in a released version of CockroachDB. The release note exists only to track the list of relevant items for the doc project.)
knz
force-pushed
the
20201218-numeric-timestamps
branch
from
40c183f
to
3605e9c
Compare
itsbilal
approved these changes
Dec 21, 2020
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, 8 of 8 files at r2.
Reviewable status:complete! 1 of 0 LGTMs obtained
|
TFYR! bors r=itsbilal |
|
Build succeeded: |
craig bot
pushed a commit
that referenced
this pull request
Dec 21, 2020
57990: eventpb: new JSON serialization with redaction markers r=itsbilal a=knz Fixes #45643 All commits but the last from #58070 and prior This is a prereq for presenting structured events "inline" in #57170. This patch introduces a new code generator and infrastructure to emit structured event payloads using the JSON syntax, but also including redaction markers for fields not marked explicitly as safe for reporting. This is not yet connected to the remainder of the logging system -- this change will be performed in a later commit. Release note: None 58015: backupccl: fix progress updating test r=pbardea a=pbardea This test was previously flaky since it was assuming that backup would issue the same number of requests as spans issued. This assumption was incorrect, and fixing the progress accounting for backup revealed that this test was faulty. While planning the work distribution for backup worker nodes, PartitionSpans automatically merges spans that are co-located on the same node, therefore reducing the number of ExportRequests issued. Before this commit, we used to block on the ExportRequest responses. The blocking was triggered on a per-range-request level. However, progress is only updated when the processor-level batch request is returned. This meant that the test might block a batch request and therefore the progress of the job would be less than what the test expected. This is now fixed by adjusting the blocking mechanism and range counting to all be at the level of the merged ranges with which the backup processor operates. Fixes #57831. This test would fail under stress pretty quickly (~40) -- I was able to get over 600 runs without failure with this patch. Release note: none 58054: opt: generate semi and anti inverted joins on multi-column indexes r=rytaft a=mgartner This commit adds tests for generating semi and anti inverted joins on multi-column inverted indexes. A `ConstFilters` field was added to `InvertedJoinPrivate`, similar to `LookupJoinPrivate` so that row count estimates for these expressions are more accurate. This was necessary to make the semi and anti joins the chosen plans for the exploration tests. A future commit will add more comprehensive stats tests. Release note: None 58081: roachtest,roachprod: Use new --log flag, fix parameterRE in expander r=itsbilal a=itsbilal Now that logging configuration is specified using --log, and not using that flag adds excess output, it's more desirable to move the disk-stalled roachtest to that flag. Also fixes a bug in the regular expression expander in roachprod, as that was conflicting with parts of YAML arguments that it shouldn't be touching. Fixes #58021. Release note: None. 58117: sql/rowexec: don't allocate buf per row in sketchInfo.addRow r=nvanbenschoten a=nvanbenschoten The `intbuf` array was meant to stay on the stack, but was escaping to the heap because the call through the `hash` function variable was opaque to escape analysis. At the end of a 4 hour, 2.2 TB IMPORT of TPC-E, this was responsible for **76.70%** of all heap allocations (by object). <img width="1684" alt="Screen Shot 2020-12-20 at 9 58 04 PM" src="https://user-images.githubusercontent.com/5438456/102735277-fb065a00-430f-11eb-837a-7ad1c903cf55.png"> 58120: storage: avoid heap allocation per value in pebbleIterator.FindSplitKey r=nvanbenschoten a=nvanbenschoten This was fallout from 95b836d. `pebble.Iterator.Value` is unsafe (no copy) but `pebbleIterator.Value` is safe (alloc + copy). ``` name old time/op new time/op delta MVCCFindSplitKey_Pebble/valueSize=32-16 97.0ms ± 5% 76.5ms ±15% -21.08% (p=0.000 n=9+10) name old speed new speed delta MVCCFindSplitKey_Pebble/valueSize=32-16 693MB/s ± 5% 881MB/s ±13% +27.19% (p=0.000 n=9+10) name old alloc/op new alloc/op delta MVCCFindSplitKey_Pebble/valueSize=32-16 27.8MB ± 0% 0.0MB ±27% -99.99% (p=0.000 n=10+10) name old allocs/op new allocs/op delta MVCCFindSplitKey_Pebble/valueSize=32-16 580k ± 0% 0k ± 0% -100.00% (p=0.000 n=10+9) ``` At the end of a 4 hour, 2.2 TB IMPORT of TPC-E, this was responsible for 15.96% of all heap allocations (by object). <img width="1680" alt="Screen Shot 2020-12-20 at 10 13 35 PM" src="https://user-images.githubusercontent.com/5438456/102735999-d6ab7d00-4311-11eb-81fb-a1bd6d807323.png"> 58121: storage: re-use buffer across calls to sstIterator.SeekGE r=nvanbenschoten a=nvanbenschoten At the end of a 4 hour, 2.2 TB IMPORT of TPC-E, this was responsible for 2.67% of all heap allocations (by object) due to the call to `sstIterator.SeekGE` from `checkForKeyCollisionsGo`. ``` File: cockroach Type: alloc_objects Time: Dec 21, 2020 at 2:51am (UTC) Duration: 5.32s, Total samples = 6963804 Active filters: focus=SeekGE Showing nodes accounting for 212999, 3.06% of 6963804 total ----------------------------------------------------------+------------- 185692 100% | github.com/cockroachdb/cockroach/pkg/storage.checkForKeyCollisionsGo /go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:3807 0 0% 0% 185692 2.67% | github.com/cockroachdb/cockroach/pkg/storage.(*sstIterator).SeekGE /go/src/github.com/cockroachdb/cockroach/pkg/storage/sst_iterator.go:93 185692 100% | github.com/cockroachdb/cockroach/pkg/storage.EncodeKey /go/src/github.com/cockroachdb/cockroach/pkg/storage/batch.go:126 ``` 58135: util: add crdb_test_off build tag to disable metamorphization r=yuzefovich a=yuzefovich We are now metamorphizing most of the test builds. However, there are cases where that behavior is undesirable (for example, when using `-rewrite` test flag with logic tests) - in such scenarios we want the "production" build to occur. In order to have such functionality this commit makes `crdb_test` build be dependent on the absence of `crdb_test_off` build tag. An example usage of `-rewrite` option would be `make testoptlogic TESTFLAGS='-rewrite' TAGS=crdb_test_off`. Release note: None Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net> Co-authored-by: Paul Bardea <pbardea@gmail.com> Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com> Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com> Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First commit from #58066.
While exploring how to use the data in practice, I noticed that the
timestamp was encoded as a string in the JSON output, and that was
making the events hard to parse and to use. There are many
more functions that can operate on a number.
Moreover, the integer representation is just more compact.
Release note (api change): The 'Timestamp' field of structured
notable events is now numeric, and encodes a number of nanoseconds
since the unix epoch. (Note that this API has not yet been published
in a released version of CockroachDB. The release note exists only to
track the list of relevant items for the doc project.)