builtins: add builtin to retrieve the payload(s) for a span. #60616

angelapwen · 2021-02-16T15:08:20Z

Part of addressing #55733

The payloads_for_span builtin retrieves all payloads for
a given span ID, given that the span is part of an active trace.
The payloads are returned in JSONB format. If the span is not
found, or if the span does not have any payloads, the builtin
returns an empty JSON object.

With the appropriate usage of this builtin and the
crdb_internal.trace_id builtin as shown in the contention_event
logic test, all payloads for the current trace may be surfaced.

Release note (sql change): add payloads_for_span builtin that
takes in a span ID and returns its paylods in JSONB format. If
the span is not found, or if the span does not have any payloads,
the builtin returns an empty JSON object.

cockroach-teamcity · 2021-02-16T15:08:35Z

This change is

60512: backupccl: update backup detached tests with retry execution under transaction r=pbardea a=Elliebababa update backup and restore detached tests with retry execution under transaction Previously, backup and restore detached tests do not retry if transactions fail. They could be flaky tests and fail in CI check. With this PR, backup and restore detached tests under transaction will not be aborted by retry errors. Release notes: None. 60594: build: explicitly add unzip as a dependency r=rickystewart a=ulfjack The `unzip` package is not installed by default in Ubuntu. It's currently pulled in as an implicit dependency of the Bazel package, but that breaks if Bazel ever removes that dependency (or when changing the Dockerfile to not install Bazel). 60648: distsql: improve test harness for processors against operators r=yuzefovich a=yuzefovich Previously, when comparing the output of processors and operators, if any order is allowed, we would perform a quadratic search. This is suboptimal, and this commit improves the situation by sorting both outputs first and then using the ordered comparison. Additionally, this commit fixes a rare flake when seemingly the same outputs would result in a mismatch. I could somewhat reliably reproduce it before the patch and cannot reproduce it after the patch. It is likely to do something with the float comparison, but I didn't get to the bottom of the flake, yet I don't think it's worth it given that this commit improves the situation anyway. Fixes: #60608. Release note: None 60666: util/tracing: re-key activeSpans map on span ID r=irfansharif a=angelapwen Note that the changes in this PR should be used in #60616 to more easily retrieve a span given its span ID, without having to visit all spans and check for a match on span ID. Whichever PR lands later should update the sections of code marked as #TODOs in #60616. Previously, a tracer's activeSpans map was keyed on the memory address of the span itself. This commit keys the map on the span ID (which is deterministically unique) and sets the corresponding value in the map to be the memory address of the span itself. This allows us to continue to visit all active spans via the map, but also easily retrieve a span from its ID using the map. Release note: None 60679: licenses: Update CCL text r=petermattis a=bdarnell Remove a dead link and clarify related text Release note: None 60681: sql: fix spacing for comments around virtual schema r=irfansharif a=irfansharif Release note: None Co-authored-by: elliebababa <ellie24.huang@gmail.com> Co-authored-by: Ulf Adams <ulfjack@users.noreply.github.com> Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com> Co-authored-by: angelapwen <angelaw@cockroachlabs.com> Co-authored-by: Ben Darnell <ben@cockroachlabs.com> Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>

irfansharif

(Holding off on further review until we introduce a tabular for for span payloads)

pkg/sql/sem/builtins/builtins.go

angelapwen · 2021-02-17T19:22:57Z

I need to rebase on top of #60666 and then push up my newest changes that use the activeSpans map rather than VisitSpans(). Please hold 😸

angelapwen · 2021-02-17T20:39:08Z

Ok, have rebased and resolved comments 🙆‍♀️ if #59992 is merged soon I'm inclined to fix up the commits here and merge what we have, then work on improving the output of the built-in to return a table separately. Otherwise I'll also do that in this PR.

irfansharif · 2021-02-17T21:07:26Z

I'd personally prefer to iterate right here, especially if we're considering changing the output form anyway. I imagine it'll end up undoing most of what was introduced in the last commit.

angelapwen · 2021-02-18T09:45:52Z

@irfansharif ah, ignore the previous comment here — I'd thought you meant iterate over the spans but you meant iterate on the PR. 😸 That sounds fine to me.

angelapwen · 2021-02-18T10:28:10Z

I've now rebased off of master now that #59992 is merged ⚡ and removed the num_payloads column of the node_inflight_trace_spans vtable that #59992 introduced.

knz

This PR is OK as a transition point. Ok to merge this after you spell out what's going to happen next.

Another confirmation that the JSON array approach is defective is that it does not perform memory monitoring and so the accumulation in RAM can cause the node to crash with an OOM error.

This would be alleviated by a streaming approach, or adding memory monitoring to the current implementation).

Reviewed 15 of 15 files at r12.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif and @tbg)

angelapwen · 2021-02-18T10:49:02Z

Thanks @knz ✨ To make sure I understand correctly, when I make the modification so that the built-in returns a table, there should not be an OOM error because the value generator functions stream values into rows default?

As for what's going to happen next:

Update the payloads_for_spans builtin to return not a JSONB Array but instead a table using the generator functions. Each row of the table would represent a single payload attached to the span. This table should include columns including at least type and JSON so that a user would be able to filter off of a certain type (or other attribute).
After this, add a pretty column to the table returned and have the JSON pretty-printed as the value.

angelapwen

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif and @tbg)

pkg/sql/sem/builtins/builtins.go, line 3611 at r11 (raw file):

Previously, angelapwen (Angela P Wen) wrote…

Not anymore! Thanks, removing locally.

Done.

pkg/sql/sem/builtins/builtins.go, line 3625 at r11 (raw file):

Previously, angelapwen (Angela P Wen) wrote…

Yep, you're right, this was a vestige of the string version we had played around with! Removing.

Done.

pkg/sql/sem/builtins/builtins.go, line 3629 at r11 (raw file):

Previously, angelapwen (Angela P Wen) wrote…

You're right, though now that I am not going to use the VisitSpans method (because I can use the activeSpans map) this is no longer relevant.

Done.

pkg/sql/sem/builtins/builtins.go, line 3634 at r11 (raw file):

Previously, angelapwen (Angela P Wen) wrote…

It doesn't ever evaluate to true — this is why I had never hit the panic when NewDJSON is passed nil yesterday. I'm removing this and in the next version where I get the span from its ID in the map, will return an empty JSON array if the span ID isn't found.

Done.

knz

add a pretty column to the table returned and have the JSON pretty-printed as the value.

thanks, can you explain this point more?

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif, @knz, and @tbg)

angelapwen · 2021-02-18T13:16:14Z

@knz I meant that the built-in's table returned could also have a column called pretty whose value is the pretty-printed version of the JSON blob (in text format). We would want both this column and the column that returns the actual JSONB value so that a user could also use the JSONB functions on this column.

By the way, I was thinking — should the builtins for trace_id and payloads_for_span require that a user is admin? If so I can add those checks to this PR.

pkg/sql/logictest/testdata/logic_test/contention_event

irfansharif · 2021-02-18T16:36:30Z

By the way, I was thinking — should the builtins for trace_id and payloads_for_span require that a user is admin? If so I can add those checks to this PR.

Yup.

The `crdb_internal.payloads_for_span` builtin retrieves all payloads for a given span ID, given that the span is part of an active trace. The payloads are returned in JSONB format. If the span is not found, or if the span does not have any payloads, the builtin returns an empty JSON object. With the appropriate usage of this builtin and the `crdb_internal.trace_id` builtin as shown in the `contention_event` logic test, all payloads for the current trace may be surfaced. Release note (sql change): add `payloads_for_span` builtin that takes in a span ID and returns its paylods in JSONB format. If the span is not found, or if the span does not have any payloads, the builtin returns an empty JSON object.

angelapwen · 2021-02-18T16:57:26Z

Added admin checks for the trace_id and payloads_for_span builtins 😺

angelapwen · 2021-02-18T18:16:48Z

bors r=knz,irfansharif

Thanks for the reviews 😄

craig · 2021-02-18T18:45:48Z

This PR was included in a batch that was canceled, it will be automatically retried

craig · 2021-02-18T20:59:46Z

Build failed (retrying...):

GitHub CI (Cockroach)

craig · 2021-02-18T22:05:02Z

Build succeeded:

GitHub CI (Cockroach)

60784: builtins: add generator builtin for span payloads r=irfansharif,knz a=angelapwen Following up from #60616. Part of addressing #55733. Previously we introduced a `crdb_internal.payloads_for_span` builtin that returned a JSONB array representing all payloads for a particular span. To improve the user experience of the builtin as well as prevent OOMs resulting from builtin usage, we replace it with a generator builtin that returns a table representing all payloads for a span, where each row represents a single payload. The new builtin, also called `crdb_internal.payloads_for_span`, has columns for the `payload_type` so that the user has the ability to easily filter on the type, and `payload_jsonb` so the user can use jsonb builtins to filter further. Release note (sql change): Update `crdb_internal.payloads_for_span` builtin to return a table instead of a JSONB array. Each row of the table represents one payload for the given span. It has columns for `payload_type` and `payload_jsonb`. Co-authored-by: angelapwen <angelaw@cockroachlabs.com>

angelapwen added the do-not-merge bors won't merge a PR with this label. label Feb 16, 2021

angelapwen requested a review from irfansharif February 16, 2021 15:08

angelapwen self-assigned this Feb 16, 2021

angelapwen force-pushed the add-payload-builtin branch from bfdc0b9 to 18dd917 Compare February 16, 2021 17:12

angelapwen changed the title ~~[WIP] Add payload builtin~~ builtins: add builtin to retrieve the payload(s) for a span. Feb 16, 2021

angelapwen removed the do-not-merge bors won't merge a PR with this label. label Feb 16, 2021

angelapwen force-pushed the add-payload-builtin branch from 18dd917 to 6dcba7d Compare February 16, 2021 17:17

angelapwen requested review from tbg and knz February 16, 2021 17:18

angelapwen force-pushed the add-payload-builtin branch from 6dcba7d to f9f0faa Compare February 16, 2021 17:41

angelapwen mentioned this pull request Feb 17, 2021

util/tracing: re-key activeSpans map on span ID #60666

Merged

angelapwen force-pushed the add-payload-builtin branch from f9f0faa to 18fbd2d Compare February 17, 2021 11:38

irfansharif reviewed Feb 17, 2021

View reviewed changes

pkg/sql/sem/builtins/builtins.go Outdated Show resolved Hide resolved

pkg/sql/sem/builtins/builtins.go Outdated Show resolved Hide resolved

pkg/sql/sem/builtins/builtins.go Outdated Show resolved Hide resolved

pkg/sql/sem/builtins/builtins.go Outdated Show resolved Hide resolved

angelapwen force-pushed the add-payload-builtin branch from 18fbd2d to 640035e Compare February 17, 2021 20:35

blathers-crl bot requested a review from irfansharif February 17, 2021 20:35

angelapwen force-pushed the add-payload-builtin branch 2 times, most recently from 33d3166 to b454338 Compare February 18, 2021 10:27

knz approved these changes Feb 18, 2021

View reviewed changes

angelapwen commented Feb 18, 2021

View reviewed changes

angelapwen force-pushed the add-payload-builtin branch from b454338 to ffe8371 Compare February 18, 2021 11:19

knz reviewed Feb 18, 2021

View reviewed changes

angelapwen force-pushed the add-payload-builtin branch from ffe8371 to a83187b Compare February 18, 2021 13:13

irfansharif reviewed Feb 18, 2021

View reviewed changes

pkg/sql/logictest/testdata/logic_test/contention_event Outdated Show resolved Hide resolved

angelapwen force-pushed the add-payload-builtin branch from a83187b to 96b24ae Compare February 18, 2021 16:56

blathers-crl bot requested a review from irfansharif February 18, 2021 16:56

irfansharif approved these changes Feb 18, 2021

View reviewed changes

craig bot merged commit d67d6cc into cockroachdb:master Feb 18, 2021

angelapwen deleted the add-payload-builtin branch February 19, 2021 09:52

angelapwen mentioned this pull request Feb 19, 2021

builtins: add generator builtin for span payloads #60784

Merged

jseldess mentioned this pull request Mar 23, 2021

builtins: add builtin to retrieve the payload(s) for a span. cockroachdb/docs#10097

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

builtins: add builtin to retrieve the payload(s) for a span. #60616

builtins: add builtin to retrieve the payload(s) for a span. #60616

angelapwen commented Feb 16, 2021 •

edited

Loading

cockroach-teamcity commented Feb 16, 2021

irfansharif left a comment

angelapwen commented Feb 17, 2021

angelapwen commented Feb 17, 2021

irfansharif commented Feb 17, 2021

angelapwen commented Feb 18, 2021 •

edited

Loading

angelapwen commented Feb 18, 2021

knz left a comment

angelapwen commented Feb 18, 2021 •

edited

Loading

angelapwen left a comment

knz left a comment

angelapwen commented Feb 18, 2021

irfansharif commented Feb 18, 2021

angelapwen commented Feb 18, 2021

angelapwen commented Feb 18, 2021

craig bot commented Feb 18, 2021

craig bot commented Feb 18, 2021

craig bot commented Feb 18, 2021

builtins: add builtin to retrieve the payload(s) for a span. #60616

builtins: add builtin to retrieve the payload(s) for a span. #60616

Conversation

angelapwen commented Feb 16, 2021 • edited Loading

cockroach-teamcity commented Feb 16, 2021

irfansharif left a comment

Choose a reason for hiding this comment

angelapwen commented Feb 17, 2021

angelapwen commented Feb 17, 2021

irfansharif commented Feb 17, 2021

angelapwen commented Feb 18, 2021 • edited Loading

angelapwen commented Feb 18, 2021

knz left a comment

Choose a reason for hiding this comment

angelapwen commented Feb 18, 2021 • edited Loading

angelapwen left a comment

Choose a reason for hiding this comment

knz left a comment

Choose a reason for hiding this comment

angelapwen commented Feb 18, 2021

irfansharif commented Feb 18, 2021

angelapwen commented Feb 18, 2021

angelapwen commented Feb 18, 2021

craig bot commented Feb 18, 2021

craig bot commented Feb 18, 2021

craig bot commented Feb 18, 2021

angelapwen commented Feb 16, 2021 •

edited

Loading

angelapwen commented Feb 18, 2021 •

edited

Loading

angelapwen commented Feb 18, 2021 •

edited

Loading