ccl/sqlproxyccl: add connector component and support for session revival token

[jaylim-crl]

Informs #76000.

Previously, all the connection establishment logic is coupled with the handler
function within proxy_handler.go. This makes connecting to a new SQL pod during
connection migration difficult. This commit refactors all of those connection
logic out of the proxy handler into a connector component, as described in the
connection migration RFC. At the same time, we also add support for the session
revival token within this connector component.

Note that the overall behavior of the SQL proxy should be unchanged with this
commit.

Release note: None

[cockroach-teamcity]

This change is 

[andy-kimball]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @jaylim-crl, @jeffswenson, and @rafiss)

---

pkg/ccl/sqlproxyccl/connector.go, line 53 at r1 (raw file):

	// startupMsg represents the startup message associated with the client.
	// This will be used when establishing a pgwire connection with the SQL pod.
	startupMsg *pgproto3.StartupMessage

NIT: I typically will capitalize fields that I expect callers to set, vs. fields that are meant to be internal to the connector (i.e. only read/written by its own methods). This makes it easier to understand which fields callers might need to initialize when construction connector.

Regardless, I think we need comments or some mechanism to know which of these fields are required and which are optional. Maybe just a comment like "This field is required."

---

pkg/ccl/sqlproxyccl/connector.go, line 87 at r1 (raw file):

	idleMonitorWrapperFn func(crdbConn net.Conn) net.Conn

	// Optional event callback functions. onLookupEvent and onDialEvent will be

NIT: Are we going too far here with this "ultra-extensible" set of callback functions? This design dates back, but I question whether we're getting much value for the extra complications it brings. Should we be inlining much/most of what these callback functions are doing?

[jaylim-crl]

TFTR!

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andy-kimball, @jeffswenson, and @rafiss)

---

pkg/ccl/sqlproxyccl/connector.go, line 53 at r1 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

NIT: I typically will capitalize fields that I expect callers to set, vs. fields that are meant to be internal to the connector (i.e. only read/written by its own methods). This makes it easier to understand which fields callers might need to initialize when construction connector.

Regardless, I think we need comments or some mechanism to know which of these fields are required and which are optional. Maybe just a comment like "This field is required."

Done. All of these fields are expected to be set by the caller, so I've capitalized all of them. Also added a comment for those.

---

pkg/ccl/sqlproxyccl/connector.go, line 87 at r1 (raw file):

Previously, andy-kimball (Andy Kimball) wrote…

NIT: Are we going too far here with this "ultra-extensible" set of callback functions? This design dates back, but I question whether we're getting much value for the extra complications it brings. Should we be inlining much/most of what these callback functions are doing?

I did this originally to avoid those logging calls within the connector component. Maybe that's unavoidable if we wanted to push everything into the connector. That said, I think removing the event callback functions are fine. We could inline everything, but I'd do this in a future PR. I'd like to push outgoingAddress into the connector, and unify this with the lookup function. The fallback onto routing rule isn't needed anymore, but we still need to support the static routing rule since a lot of tests rely on that. I added a TODO here.

One of the goals with this PR is to ensure that the behavior of SQL proxy remains unchanged.

[jeffswenson]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andy-kimball, @jaylim-crl, and @rafiss)

---

pkg/ccl/sqlproxyccl/connector.go, line 87 at r1 (raw file):

Previously, jaylim-crl (Jay Lim) wrote…

I did this originally to avoid those logging calls within the connector component. Maybe that's unavoidable if we wanted to push everything into the connector. That said, I think removing the event callback functions are fine. We could inline everything, but I'd do this in a future PR. I'd like to push outgoingAddress into the connector, and unify this with the lookup function. The fallback onto routing rule isn't needed anymore, but we still need to support the static routing rule since a lot of tests rely on that. I added a TODO here.

One of the goals with this PR is to ensure that the behavior of SQL proxy remains unchanged.

+1 to replacing the indirection with direct calls and pulling the logging and metrics into the connector. I think it is fine to accept the things that are already callbacks, e.g. the testingKnobs struct. The new callbacks are the ones I would prefer to see removed from the PR. E.g AddrLookupFn, IdleMonitorWrapperFn, AuthenticateFn, OnLookupEvent, and OnDialEvent.

[jaylim-crl]

TFTR! Addressed all the reviews. This PR is ready for another round of review.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andy-kimball and @rafiss)

---

pkg/ccl/sqlproxyccl/connector.go, line 87 at r1 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

+1 to replacing the indirection with direct calls and pulling the logging and metrics into the connector. I think it is fine to accept the things that are already callbacks, e.g. the testingKnobs struct. The new callbacks are the ones I would prefer to see removed from the PR. E.g AddrLookupFn, IdleMonitorWrapperFn, AuthenticateFn, OnLookupEvent, and OnDialEvent.

Done. I backed out most of the changes except IdleMonitorWrapperFn, which was also discussed offline. We can't move that in because the existing monitor takes a callback which contains a reference to errConnection (and I didn't want to move that channel into the connector):

cockroach/pkg/ccl/sqlproxyccl/proxy_handler.go

Lines 396 to 402 in 07a3683

	crdbConn = handler.idleMonitor.DetectIdle(crdbConn, func() {
	err := newErrorf(codeIdleDisconnect, "idle connection closed")
	select {
	case errConnection <- err: /* error reported */
	default: /* the channel already contains an error */
	}
	})

As for metrics, I don't think we should do that right now. I'll go through the proxy once the connection migration stuff is in to start collecting metrics for items that we care about. I suspect there are some unused/not useful metrics at the moment as well. I moved the logging stuff into the connector though.

[jeffswenson]

LGTM

I like the direction you are taking the sqlproxy. proxy_handler was a bit much. Pulling logic out into the connector and forwarder is a big improvement.

[jaylim-crl]

TFTR!

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andy-kimball, @jeffswenson, and @rafiss)

---

pkg/ccl/sqlproxyccl/connector.go, line 197 at r3 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

nit: consider naming openClusterConnInternal "dialTenantCluster".

Done.

---

pkg/ccl/sqlproxyccl/connector.go, line 278 at r3 (raw file):
The latter would be tricky to test I think. I went with the first approach. Did not log here because we're already logging in the caller.

avoid panics in the data plane code

I have mixed feelings about this. Back then when I returned errors, people told me they preferred panics especially when a case is impossible; it's better to crash right away than letting it go as an error to the caller. I personally prefer the panic approach, but I've changed this anyway. If you grep for panic("unreachable"), you'll find some in the kv or sql layer.

---

pkg/ccl/sqlproxyccl/connector.go, line 334 at r3 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

nit: consider naming dialOutgoingAddr "dialSqlServer(serverAddr string)"

Done.

[jaylim-crl]

bors r=JeffSwenson

[craig]

Build failed (retrying...):

• GitHub CI (Cockroach)

[craig]

Build succeeded:

• GitHub CI (Cockroach)