go/adbc/driver/flightsql: add extra logging

[lidavidm]

grpc-go doesn't log outgoing headers - ideally we should have a way to enable logging of headers + Flight/Flight SQL operation for debugging

• Incoming, outgoing headers (optionally, since this may leak credentials)
• Incoming, outgoing headers (header names only)
• Requests made to each server, and server locations (and semantically, 'which connection' the requests correspond to)
• Bake in some understanding of OpenTelemetry?
• The raw gRPC error

[rtadepalli]

Task 1 seemed straightforward so I tried to log the headers by registering an interceptor (similar to unaryTimeoutInterceptor) while creating the flight client and then it occurred to me -- should this logging happen in the arrow flight client library instead of within flightsql_adbc.go? May be easier to get the headers there, but I might be missing something.

Also, there's no logging package present in go.mod, so I am not sure if there is a preference for which package to add or if we should just write to stdout. Thanks.

[lidavidm]

Possibly. @zeroshade opinions?

That said it would be an interceptor either way - we can always refactor or upstream things and I think we can iterate more quickly here.

I'm not sure about the state of Go logging. Ideally a structured logging package, compatible with OpenTelemetry (but configurable via environment variable for easy debugging), would be used but I'm not sure if that's available.

[zeroshade]

So the base Arrow client flight library has a function that can be used to easily construct interceptors from simple interfaces to grab the headers. It's pretty easy to create a middleware to perform logging using these interfaces.

As far as logging in Go, for most cases the standard logging package is more than good enough. But if you want more structured logging zap, zerolog, and apex are the most popular structured logging libraries currently.

For something like ADBC, I'd say we're probably best off setting up a logging interface to allow users to plug whatever logger they want (i.e. follow the stdlib logger's interface because most logging libraries support that interface for compatibility).

For OpenTelemetry, we definitely need to add otel instrumentation to the base arrow flight and flightsql packages, it's not hard. The package is very good and we already pass the context around which would contain the necessary headers etc. It's on my list of things to do, but I'd highly welcome and review any PRs for it!

[lidavidm]

Punting on this for the next release, but @zeroshade are you suggesting we essentially add to the ADBC interface so you can set a logger on the Database object (or similar)? (And then we can figure out how to bind that through to say C/C++ and so on; possibly we want a SetOptionFuncPointer or something?)

[lidavidm]

It looks like slog has a backport, so we could define the extension interface, log to a driver-wide logger, and have the FFI part configure the logger (using some standard environment variables) for debugging.