access log: support generic protocol

[caitong93]

Title: access log: support generic protocol

Description:

I want to add access log support for dubbo and thrift filter, but find out current access log formatter only support TCP / HTTP, and it cannot be extended. Any thoughts about implement a generic access log formatter ?
Thanks !

[mattklein123]

See

envoy/api/envoy/config/accesslog/v3/accesslog.proto

Line 318 in 308619f

message ExtensionFilter {

[caitong93]

ExtensionFilter seems not what I'm looking for. Envoy::AccessLog::Instance, Envoy::AccessLog::Formatter, Envoy::AccessLog::FormatterProvider, and implementations of these interfaces, only support TCP / HTTP.

[lizan]

This seems a valid feature request but need design proposal. @mattklein123 the extension filter is about how we filter log entries, but it doesn't let access log support generic protocol, other than TCP / HTTP.

@cpakulski was looking into how to integrate MySQL logs (e.g. transaction logs which is in different layers than TCP), we saw same issue with current logging structure including ALS. It makes sense to have some extensible way to let TCP filters to emit custom logs easily (to access logger extensions, file/gRPC/etc).

[mattklein123]

It makes sense to have some extensible way to let TCP filters to emit custom logs easily (to access logger extensions, file/gRPC/etc).

I agree. We actually need access logs for the DNS filter also at Lyft (cc @abaptiste) so whatever we come up with here can be used for that.

Naively I would think that the general HTTP/TCP system should work with possible protocol specific data extensions?

[cpakulski]

I am working on a prototype. The idea is to emit fairly flexible messages describing non-HTTP transactions, like database interactions. I could not fit them into gRPC message, as it is very HTTP specific.

[caitong93]

add ref to #14175 , which might be related to this issue

[wbpcode]

Is there any progress here ? GRPC does make this problem very complex. Perhaps we could start by creating a generic file access logger for none HTTP protocol and gradually extend it to other access loggers such as GRPC.

@cpakulski @lizan @mattklein123