Ensure pprof endpoint is listening on startup

[jsoriano]

pprof HTTP endpoint was being initialized asynchronously. In tests we
have found that sometimes after Metricbeat has started, it fails to
connect to the pprof endpoint. This can happen if the server is not
listening before the rest of the beat is initialized.

This change ensures that the listener for this endpoint is started
before continuing with the rest of the initialization. This should reduce
flakiness of tests that depend on these endpoint (test_stats tests).

Some comments and error messages have been polished.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[blakerouse]

lgtm!

[jsoriano]

@blakerouse sorry, I have added a couple more changes, could you please take another look?

Doing some tests I found that fatal errors in this code were not being logged, I think that because of the use of log instead of libbeats logp. I have changed the calls to log.Fatal by calls to logp.Error + os.Exit and they are logged now.

I have also added a changelog entry because before this change if httpprof couldn't be started it was being mostly ignored (it only logged something like this at the info level):

2020-03-16T16:37:54.167+0100	INFO	service/service.go:112	finished pprof endpoint: listen tcp 127.0.0.1:22: bind: permission denied

[blakerouse]

I think the goal is to move away from the global logger, shouldn't log.NewLogger be used instead of the global logp logger?

[jsoriano]

I think the goal is to move away from the global logger, shouldn't log.NewLogger be used instead of the global logp logger?

Yes, you are right. I have tried to do it, but this package is formed of global methods called from beat initialization code, that is still using the global logger.
I have tried to define a global logger in this package but it didn't log anything. At the end I am instantiating a new logger on every method that needs it. I am not very happy with this change but it is some progress, at least it is using the per-instance methods.
Maybe it'd be better to move this change to its own PR, I am not sure what are the expectations regarding logging for this initialization code.
Let me know what you think.

[blakerouse]

Looks good.

Agree that creating a logger for each function is rather weird, but no real other alternative.