Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colexec, coldata: add support for INTERVAL type #43517

Merged
merged 2 commits into from
Jan 30, 2020

Conversation

yuzefovich
Copy link
Member

@yuzefovich yuzefovich commented Dec 24, 2019

pgerror: clean up build deps

The pgerror (and pgcode) packages are (perhaps inadvisably) used in
low-level utility packages. They had some pretty heavyweight build deps,
but this wasn't fundamentally necessary. Clean it up a bit and make
these packages more lightweight.

Release note: None

colexec, coldata: add support for INTERVAL type

This commit adds the support for INTERVAL type that is represented by
duration.Duration. Only comparison projections are currently supported.
The serialization is also missing.

Addresses: #42043.

Release note: None

@yuzefovich yuzefovich requested review from asubiotto and a team December 24, 2019 05:50
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@yuzefovich yuzefovich force-pushed the vec-interval branch 2 times, most recently from b503a6e to f4121eb Compare December 31, 2019 04:45
@yuzefovich yuzefovich requested a review from a team as a code owner December 31, 2019 04:45
@yuzefovich yuzefovich force-pushed the vec-interval branch 3 times, most recently from bddde60 to ced4547 Compare December 31, 2019 19:44
Copy link
Contributor

@asubiotto asubiotto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: Do we have an issue that tracks types that need serialization/more efficient serialization? We should also track the need to support other types of operators for certain types, including interval.

duration.Duration doesn't lend itself to efficient serialization, but I'm not sure what to do about it right now. We might need to write an abstraction layer on top of certain types that have inefficient serialization at some point. This might be something equivalent to flat bytes where we represent []duration.Duration as a slice of int64s that are interpreted into durations three by three.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @asubiotto)

@yuzefovich
Copy link
Member Author

@danhhz are you ok with the second commit here which whitelists many dependencies that util/duration brings into pkg/workload?

@yuzefovich yuzefovich force-pushed the vec-interval branch 3 times, most recently from 92e0cd6 to 97ac5ff Compare January 22, 2020 05:08
@yuzefovich
Copy link
Member Author

Friendly ping @danhhz

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

Woah, that's a lot of stuff! No we probably shouldn't do that. Have you looked into why it brings so much in?

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

It looks like the issue is the errEncodeOverflow error returned from Duration.Encode. I haven't looked closely, but it may be an easy fix for that method to return a bool (true for overflow) instead of an error, and turn that into a errEncodeOverflow at the callers

@yuzefovich
Copy link
Member Author

Thanks for the comments, I'll take a deeper look.

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

So I tried prototyping this and quickly found out that this error ends up getting plumbed back all the way even through the KV api (via internal/client/util.marshalValue). I think the issue here is that the pgcode error wrapper, while useful context for the user, has been sprinkled in utility libraries that it probably doesn't belong in. All of this is not your code, I understand. Hmmm....

To be honest, how much do we lose by simply removing the pgcode wrapper from this one place? That certainly seems like the easiest fix. We still get the error message for the user and I somewhat doubt any clients are actually sniffing this code to do something useful

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

The other way we could go at this is to make pgerror a lighter dep.

  • The real issue seems to be its use of github.com/cockroachdb/cockroach/pkg/util/log, which is only included for DecodeStackTrace and EncodeStackTrace, both of which only have stdlib and external deps. So we could move those two somewhere else.
  • The other one seems to be the decodeWithCandidateCode method, which has an argument of type "protoutil.SimpleMessage". This appears to be the case only because having "proto.Message" would be a linter error, so maybe we can disable that one lint here

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

I'm leaning toward the last one. It will be functionally identical. Plus, while in theory I think putting these pgerrors in util packages is unfortunate, I must admit that practically it's not worth it to fix them at this point. Cleaning up pgerror lets us sidestep the issue

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

Ugh, just noticed that DecodeStackTrace uses log.Errorf

@danhhz
Copy link
Contributor

danhhz commented Jan 27, 2020

Okay, I think I made this work. @yuzefovich would you mind cherry-picking this into your PR? https://github.com/cockroachdb/cockroach/compare/master...danhhz:pgerror_deps?expand=1

@yuzefovich
Copy link
Member Author

Yeah, that seems to do the trick, thanks!

@yuzefovich yuzefovich force-pushed the vec-interval branch 2 times, most recently from 00c5b52 to 0bfa84e Compare January 27, 2020 23:56
@yuzefovich
Copy link
Member Author

Updated, PTAL.

@danhhz
Copy link
Contributor

danhhz commented Jan 28, 2020

LGTM

@yuzefovich
Copy link
Member Author

I needed to make a few tweaks to the commit that cleans up the dependencies, please take another look.

@yuzefovich
Copy link
Member Author

@danhhz could you please take another quick look that the changes I made to make the linters happy are acceptable.

Copy link
Contributor

@danhhz danhhz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops sorry! This got lost in my inbox. LGTM

danhhz and others added 2 commits January 30, 2020 08:38
The pgerror (and pgcode) packages are (perhaps inadvisably) used in
low-level utility packages. They had some pretty heavyweight build deps,
but this wasn't fundamentally necessary. Clean it up a bit and make
these packages more lightweight.

Release note: None
This commit adds the support for INTERVAL type that is represented by
duration.Duration. Only comparison projections are currently supported.
The serialization is also missing.

Release note: None
@yuzefovich
Copy link
Member Author

TFTRs!

Do we have an issue that tracks types that need serialization/more efficient serialization?

Yep, filed #44195.

We should also track the need to support other types of operators for certain types, including interval.

So far (I think) the only missing projections are of combination of Timestamps and Intervals. There is an issue for that #42044.

bors r+

craig bot pushed a commit that referenced this pull request Jan 30, 2020
43517: colexec, coldata: add support for INTERVAL type r=yuzefovich a=yuzefovich

**pgerror: clean up build deps**

The pgerror (and pgcode) packages are (perhaps inadvisably) used in
low-level utility packages. They had some pretty heavyweight build deps,
but this wasn't fundamentally necessary. Clean it up a bit and make
these packages more lightweight.

Release note: None

**colexec, coldata: add support for INTERVAL type**

This commit adds the support for INTERVAL type that is represented by
duration.Duration. Only comparison projections are currently supported.
The serialization is also missing.

Addresses: #42043.

Release note: None

Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
@craig
Copy link
Contributor

craig bot commented Jan 30, 2020

Build succeeded

@craig craig bot merged commit f7211dd into cockroachdb:master Jan 30, 2020
@yuzefovich yuzefovich deleted the vec-interval branch January 30, 2020 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants