Add embedded Go pprof and DataDog profilers

[rdmitr]

This PR replaces #320 — I got linter errors when trying to merge and commit, so I figured it was easier to ditch it and create a new one 😅

[rdmitr]

@bumi in the original PR, you have removed all the env configuration variables except DD_PROFILER_AGENT_ADDR. I am a bit confused — was it intended?.. Agent address can be specified using an env variable just like the other DataDog settings, so in this PR I have added DD_PROFILER_ENABLED flag instead.

[rolznz]

@rdmitr do you remember the linter errors you got? we merged some linter changes but it should have only affected the frontend code, not the go code.

[bumi]

@rdmitr makes sense. thanks

[rdmitr]

@rolznz

do you remember the linter errors you got? we merged some linter changes but it should have only affected the frontend code, not the go code.

Not really — I think some of the commit names were not formatted as expected by husky, and then it screwed up the merge and conflict resolution process in my IDE. I decided not to waste time trying to figure it out.

But the husky rules certainly apply to the whole repo — I was quite surprised on Saturday when my commits suddenly started failing 😁

[rolznz]

@rdmitr do you understand what this part is caused by?

Can we conclude that LDK consumes about 4X as much resources as our NWC code and our main two things that cause high CPU usage are Nostr event handling and DB access?

[rolznz]

tACK for pprof.

We still need to deploy a datadog agent and enable profiling on some of our running hub instances.

[rdmitr]

@rolznz

do you understand what this part is caused by?

As far as I understand, that's goroutine management code

Can we conclude that LDK consumes about 4X as much resources as our NWC code and our main two things that cause high CPU usage are Nostr event handling and DB access?

It looks like the profile was collected over a fairly short amount of time, so it's hard to say for sure. But yes, I would expect LDK code to be quite resource intensive