Log reconnect attempts

[recrsn]

Closes #5175

Logs an info line "Attempting to reconnect" at every reconnect attempt.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[kvch]

jenkins test it

[urso]

+1 for logging connection and reconnect attempts

We have two loops in this methods. First loop attempts to connect. Second loop processes events.

The first loop becomes active after beats startup, once the first event is to be published. That is connecting the outputs is done lazily. In this case the log message should indicate this being the initial connect.

Once the send loop fails, we're about to reconnect.

Having a counter on failed connect attempts would be nice to have as well.

E.g. for the lifetime of an output it would be nice to have log messages like (assuming we would have an 'identifier' per output):

Startup output to <identifier>
Connect to <identifier>
Failed to connect to <identifier>: <reason>
Connect to <identifier>
Connection to <identifier> established
Failed to publish events to <identifier>: %v
Closed connection to <identifier>
Attempting to reconnect to <identifier> with 0 reconnect attempt(s).
Failed to connect to <identifier>: <reason>
Attempting to reconnect to <identifier> with 1 reconnect attempt(s).
Connection to <identifier> established
Closed connection to <identifier>
Shutdown output to <identifier>

That is, we have 4 phases/states to distinguish:

1. init worker
2. create initial connection
3. reconnect
4. shutdown

Having an enum to differentiate on state and logging should help here.

[recrsn]

I wanted to use hostname as an identifier. However, we don't have it here.
I'll be adding additional logging lines for other events.

[recrsn]

@urso We counts for reconnect attempts now.

[recrsn]

@urso I need some pointers for how to create the identifier.

The state enum should probably go into the Connection type ?

[ruflin]

@agathver The team is at Elastic{ON} / All hands this and next week so it will take some time to respond.

[recrsn]

Any updates on this?

[urso]

The counting looks fine.

It would be nice to have an identifier in general. It's the client knowing about the actual endpoint/ID in use. Let's add the identifier to the output.Client.

type Client interface {
        String() string // alternative names: ID(), Name(), Endpoint(), Target()
	Close() error
	Publish(publisher.Batch) error
}

I used the String() method, as this way the client can be pushed as parameter to the logging functions.

The console output will return "stdout", the file output the path and the network based outputs the url/host/port.

[recrsn]

@urso, Added identifiers for connections.

I have used a composition-like approach for naming (borrowed from the convention in ReactJS), in tune with how most clients are being used.

Example, a redis client wrapped with a failover will be failover(redis://10.0.0.1, redis://10.0.0.2)
Default ES config gives backoff(elasticsearch(http://localhost:9200))

[urso]

Great improvement. Sorry for having you wait. Thank you for taking the time to add these!