server: use a shared HTTP listener for all in-memory tenant servers

[knz]

Parent PRs:

• server: enabling instantiating secondary tenant servers in-memory  #84608
• server,netutil: clean up the connManager logic #91739

Fixes #91738.

This commit introduces a HTTP (de)multiplexer for all in-memory tenant
servers.

By default, HTTP requests are routed to the system tenant server.
This can be overridden with the header X-Cockroach-Tenant or the tenant cookie.

Epic: CRDB-14487

[cockroach-teamcity]

This change is 

[knz]

@dhartunian do you want to have a preliminary look to check if the overall direction is satisfactory?

I'm still on the fence about whether to select the tenant via a custom header or a cookie. Maybe we should do both? Let me know what you think.

[dhartunian]

I'm still on the fence about whether to select the tenant via a custom header or a cookie. Maybe we should do both? Let me know what you think.

I like supporting multiple methods. Cookie is best for the frontend since it allows all requests to flow reliably to the right place. For instance, requests to JS/CSS/images etc. can't have the header attached because the browser makes them. The only way to ensure they come from the correct tenant is to use a cookie. I know we're not serving different assets at the moment, but it's nice to not worry about it.

Reviewed 11 of 11 files at r1, 12 of 12 files at r2, 5 of 5 files at r3, 4 of 7 files at r4, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @herkolategan, @knz, and @srosenberg)

---

pkg/server/server_controller.go line 174 at r4 (raw file):

// the special HTTP request header.
// If no tenant is specified, the default tenant is used.
func (c *serverController) httpMux(w http.ResponseWriter, r *http.Request) {

love how simple this ends up being!

---

pkg/server/server_controller.go line 277 at r4 (raw file):

func (t *systemServerWrapper) getHTTPHandlerFn() http.HandlerFunc {
	return t.s.http.baseHandler

nit: can we standardize between the t.s and t.server names here?

[knz]

I like supporting multiple methods.

👍 Done.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dhartunian, @herkolategan, and @srosenberg)

---

pkg/server/server_controller.go line 277 at r4 (raw file):

Previously, dhartunian (David Hartunian) wrote…

nit: can we standardize between the t.s and t.server names here?

Done.

[dhartunian]

Reviewed 1 of 18 files at r5, 18 of 18 files at r9, 12 of 12 files at r10, 5 of 5 files at r11.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @herkolategan, @knz, and @srosenberg)

---

pkg/server/server_controller.go line 326 at r12 (raw file):

	// Start a goroutine that watches for shutdown requests issued by
	// the server itself.
	go func() {

are we trying to avoid go keyword in non-test code?

[knz]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dhartunian, @herkolategan, and @srosenberg)

---

pkg/server/server_controller.go line 326 at r12 (raw file):

Previously, dhartunian (David Hartunian) wrote…

are we trying to avoid go keyword in non-test code?

It's a good question. Alas we can't use stopper.RunAsyncTask here because the call to stopper.Stop inside would then deadlock. Clarified in comment.

[andreimatei]

I don't have opinions on cookie vs header that you're discussing. However, what I know how to edit is the URL in the address bar. Could we also take the tenant there, like we do with the node id for proxying HTTP requests between nodes?

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @dhartunian, @herkolategan, @knz, and @srosenberg)

---

pkg/server/config.go line 214 at r16 (raw file):

	StartDiagnosticsReporting bool

	// DisableOwnHTTPListener prevents this server from starting a TCP

nit: wrap

---

pkg/server/config.go line 216 at r16 (raw file):

	// DisableOwnHTTPListener prevents this server from starting a TCP
	// listener for the HTTP service. Instead, it is expected
	// that some other service (typically, the server controller)

s/server controller/serverController

---

pkg/server/config.go line 218 at r16 (raw file):

	// that some other service (typically, the server controller)
	// will accept and route requests instead.
	DisableOwnHTTPListener bool

s/DisableOwnHTTPListener/DisableHTTPListener

---

pkg/server/server_controller.go line 326 at r12 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

It's a good question. Alas we can't use stopper.RunAsyncTask here because the call to stopper.Stop inside would then deadlock. Clarified in comment.

but the comment was added in the wrong commit :P

---

pkg/server/server_controller.go line 188 at r16 (raw file):

	}
	if tenantName == "" {
		// TODO(knz): Make the default tenant route for HTTP configurable.

Take it or leave it but, in my opinion, TODOs like this serve little purpose other than distracting and making reading the code harder. There's no need for this code to anticipate what might or might not come in the future; the code makes enough sense as it is (TODOs are mostly for when the code makes less than enough sense as it is).

---

pkg/server/server_controller.go line 194 at r16 (raw file):

	s, err := c.get(ctx, tenantName)
	if err != nil {
		log.Warningf(ctx, "unable to find tserver for tenant %q: %v", tenantName, err)

tserver ?

---

pkg/server/server_controller.go line 194 at r16 (raw file):

	s, err := c.get(ctx, tenantName)
	if err != nil {
		log.Warningf(ctx, "unable to find tserver for tenant %q: %v", tenantName, err)

s/unable to find/failed to start ?

---

pkg/server/server_controller.go line 206 at r16 (raw file):

// Used in tests until https://github.com/cockroachdb/cockroach/issues/84585 is resolved.
func (s *Server) TestingGetSQLAddrForTenant(tenant string) string {
	ts, err := s.serverController.get(context.Background(), tenant)

nit: take in a ctx

---

pkg/server/tenant.go line 330 at r16 (raw file):

	// If DisableOwnHTTPListener is set, we are relying on the HTTP request
	// routing performed by the server controller.

s/server controller/serverController

[dhartunian]

thanks for adding the query param option.

Reviewable status: complete! 1 of 0 LGTMs obtained (and 1 stale) (waiting on @andreimatei, @herkolategan, and @srosenberg)

[knz]

TFYR!

bors r=andreimatei,dhartunian

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)