-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Async logging #988
Comments
My use case is the one mentioned here |
Hello! Yes, depending on your needs, you should be able to use the recently added zapcore.BufferedWriteSyncer to write your logs to disk asynchronously. Hope this helps! Closing since the question was answered. |
zap/zapcore/buffered_write_syncer.go Line 107 in c8e813e
It does not seem to be async if you see above we are using mutex locks Is there any other alternatives? |
Why not use channels with a large/configurable size? That way, writes happen to channel from the calling goroutine and another goroutine reads data and sends it to the underlying writer which can be a buffered_write_syncer to have fine grained control of the flush |
The mutex inside BufferedWriteSyncer does not change whether logging is happening asynchronously here. Sorry, let me clarify my understanding of the problem, and why I think BufferedWriteSyncer solves that. The question is whether there is a way to log with Zap in such a way that disk IO, i.e. writing the logs to a file, does not happen on the request path. We want it so that the following statement does not perform any disk IO.
Instead, we want the statement to record the log message in-memory, and then we want a separate goroutine to write to disk out of band. This will have the effect of not blocking the main goroutine on IO. And that's exactly what BufferedWriteSyncer will do: it will write to an internal buffer, flushing to the file as needed in a separate goroutine. I'll grant that this isn't 100% asynchronous because if logs are written at a rate faster than flush can keep up, like if there's a sudden spike, then one of the log calls will wait for the BufferedWriteSyncer to flush. |
My ideal asynchronous log writing is a goroutine that processes the asynchronous log from the channel all the time, and the goroutine blocks when there is no data in the channel. Of course, this channel has a certain capacity or memory capacity setting. In this way, there will indeed be blocking when a large number of logs are written, but when there is no log writing, the goroutine is blocked. The main reason is that I want to write logs to the console and to a file at the same time, I think BufferedWriteSyncer is really great for writing logs to a file periodically. But for me to set an infinitely small FlushInterval in order to print the log on the console almost all the time, I feel that BufferedWriteSyncer is not very suitable. Looking forward to your reply, thank you. 谢谢! Here is an example of what I want to achieve, relatively simple buffered channel implementation for go 1.18 var (
sugarLogger *zap.SugaredLogger
loggerChannel = make(chan *logItem, 5000)
)
func InitLogger(opt Options) error {
encoderConfig := zap.NewProductionEncoderConfig()
encoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
encoderConfig.EncodeLevel = zapcore.CapitalLevelEncoder
encoder := zapcore.NewJSONEncoder(encoderConfig)
logWriter := zapcore.AddSync(&lumberjack.Logger{
Filename: opt.LogFileName,
MaxSize: opt.MaxSize,
MaxAge: opt.MaxAge,
MaxBackups: opt.MaxBackups,
LocalTime: false,
Compress: false,
})
// string to level
var l zapcore.Level
if err := l.Set(opt.Level); err != nil {
return err
}
// 同时打印到控制台和文件
core := zapcore.NewCore(encoder, zapcore.NewMultiWriteSyncer(zapcore.AddSync(os.Stdout), logWriter), zap.NewAtomicLevelAt(l))
logger := zap.New(core, zap.AddCaller(), zap.AddCallerSkip(1), zap.AddStacktrace(zap.ErrorLevel))
sugarLogger = logger.Sugar()
go func() {
for {
select {
case log := <-loggerChannel:
logByLevel(log.level, log.template, log.args)
}
}
}()
return nil
}
type logItem struct {
args []any
template string
level zapcore.Level
ctx context.Context
}
func Info(template string, args ...any) {
loggerChannel <- &logItem{level: zapcore.InfoLevel, template: template, args: args}
}
func Error(template string, args ...any) {
loggerChannel <- &logItem{level: zapcore.ErrorLevel, template: template, args: args}
}
func logByLevel(level zapcore.Level, template string, args []any) {
switch level {
case zapcore.DebugLevel:
sugarLogger.Debugf(template, args...)
case zapcore.InfoLevel:
sugarLogger.Infof(template, args...)
case zapcore.WarnLevel:
sugarLogger.Warnf(template, args...)
case zapcore.ErrorLevel:
sugarLogger.Errorf(template, args...)
case zapcore.DPanicLevel:
sugarLogger.DPanicf(template, args...)
case zapcore.PanicLevel:
sugarLogger.Panicf(template, args...)
case zapcore.FatalLevel:
sugarLogger.Fatalf(template, args...)
}
} |
Adding to the above, here is another example of the same with next level composability from log15 library |
Hi,
Here's the sink code :
|
Does zap support async logging?
The text was updated successfully, but these errors were encountered: