Slow performance

[coder-mike]

I wanted to test out this library with DataDog. I wanted to evaluate the performance under load.

I basically copied the example code and then got it to spit out 10,000 log entries.

var winston = require('winston')
var DatadogWinston = require('datadog-winston')

var logger = winston.createLogger({})

logger.add(
  new DatadogWinston({
    apiKey: '...',
    hostname: 'myservice',
    service: 'super_service',
    ddsource: 'nodejs',
    ddtags: 'foo:bar,boo:baz'
  })
)

for (let i = 0; i < 10_000; i++)
  logger.info('Hello, World!')

What I'm seeing in the DataDog UI is that these 10,000 log entries trickle through over the course of 8 minutes, which is a measly 21 entries per second. I would expect orders of magnitude more than this.

I don't know if I'm just approaching this whole thing the wrong way, if there's another recommended way to get node.js log entries into Datadog, or if I'm supposed to be batching these somehow. Could someone please advise?

[itsfadnis]

Thanks for raising the issue @coder-mike. You're right, the performance is really bad under load. The library fires an http request per log entry which is not efficient. The recommended way to collect logs is via the datadog agent. I'd recommend you going with it instead of this library. Refer https://docs.datadoghq.com/logs/log_collection/nodejs/?tab=winston30.

[coder-mike]

The agent would also be making network requests to submit the logs to the DataDog server using some protocol. If that protocol is apparently more efficient, is it difficult for this library to support that protocol?

I don't really see the theoretical advantage of trying to speed up communication to a remote service by adding an extra middleman (agent) which is essentially tasked with the same objective (communicate to the remote service).

I ask because having an agent requires having access to that layer of the VM, which then loses platform independence and makes deployment more difficult in a FaaS or other managed environment. This is illustrated by the fact that the documentation you linked talks about modifying conf.d/, which is an instruction that only works in a subset of environments (only Linux environments and only environments where we have access to this file).

[itsfadnis]

@coder-mike I'm not sure how the datadog agent works, so not sure what protocol it uses or if it is faster. But yes installing another agent just to forward logs is not ideal.

From the perspective of this library I can think of two improvements:

1. Introduce log compression while sending over the network
2. Log batching and flushing them at regular intervals

I think these improvements can bring in better performance.

[coder-mike]

I think batching would make a big difference, if the DataDog HTTP protocol supports that.

[coder-mike]

I closed the ticket accidentally.

[hrvojepavlinovic]

Here it says that logs can be sent in arrays with up to 1000 entries: https://docs.datadoghq.com/api/latest/logs/#send-logs and according to Winston docs, we can define batching options for HTTP transport: https://github.com/winstonjs/winston/blob/master/docs/transports.md#http-transport, so in theory, we can send logs in batches, but at least in my project, Winston 3.3.3 typings doesn't have batch config properties.

EDIT: I've found related PR, batching support should be included in next Winston release: winstonjs/winston#1970

[JCMais]

any updates on this?

[udbmnm]

any updates on this?

https://github.com/winstonjs/winston/blob/master/docs/transports.md#http-transport
You can use batch to control whether to send data in bulk or not