Bug: Caddy freezes when syslog server not available

[sopleb]

Hello,

I had a power outage over the weekend and noticed an issue with one of my sites, when looking into the issue, caddy was locking up whenever browsing the one domain that uses a syslog output

somewebsite.com {
        reverse_proxy host:port
        log {
                output net syslog-server:port
        }
}

Is the config that is being used. Is this behavior expected? or am I breaking something?

[simaotwx]

I'm not super familiar with the Caddy codebase but here's what I'm seeing:

In https://github.com/caddyserver/caddy/blob/master/modules/logging/netwriter.go#L78 it attempts to connect to the server which causes Caddy to lock up.
Called from https://github.com/caddyserver/caddy/blob/master/logging.go#L656
Which is called from https://github.com/caddyserver/caddy/blob/master/logging.go#L147
Which is called from https://github.com/caddyserver/caddy/blob/master/logging.go#L102
Which in turn is called from https://github.com/caddyserver/caddy/blob/master/caddy.go#L360

I'm not sure how to solve this. We could use a goroutine to set up logging and use the default logger until then but that could cause some messages to get lost. Or we use a context deadline but that wouldn't solve the downtime issue.
Maybe a combination of both could be the solution: waiting a bit and if nothing happens, use the the default logger until the connection succeeds.

I think the others can give more insight into this.

[mholt]

I think the best solution would be to not configure Caddy to use a log server that isn't available?

[v-rosa]

@mholt considering that initially the syslog server is available, but suddenly its not reachable, this problem will persist and impact the caddy server, right?

[mholt]

I haven't had much time to spend on this, but I pushed a potential patch in #4172 -- could you all please try it out? Configure a dial timeout and it should unblock the server so it can keep working, it just might still be slow (once in a while). Let me know the details of your experience with it and we can smooth out any rough edges.