Add totalsize metric in MongoDB collstats metricset

[ishleenk17]

Describe the enhancement:

Add the totalsize metric to mongodb collstats.
This is an important metric to report the total size of the collection.

Once this is added to the beats module, we should add the same to Integrations datastream as well.

[ishleenk17]

We should revisit all the collstats metrics as I think we should add couple other important metrics as well.
The list of the metrics available .

[stefans-elastic]

This confuses me a bit: it looks like collstats module doesn't actually return collstats data. It return results of top command: https://github.com/elastic/beats/blob/main/metricbeat/module/mongodb/collstats/collstats.go#L79
Also if we are going to fetch collstats metrics I think we need to call collstats command on each collection (which is likely going to degrade performance of the module). Are we ok with this?

[ishleenk17]

That's right, this is what I noticed as well while checking the beats code. That we are returning just the usage metrics for the collections from the top command. Listed here as well. We are not having any other metrics. Hence I pointed out we might want to revisit the complete collstats module.

Also if we are going to fetch collstats metrics I think we need to call collstats command on each collection (which is likely going to degrade performance of the module)

I think we don't have any way other than using the collstats command, is that right ?

[stefans-elastic]

I think we don't have any way other than using the collstats command, is that right ?

That's my understanding (I haven't found any other way)

[stefans-elastic]

@ishleenk17 I've come up with a "PoC" (kinda): eb7bbcb

If this solution makes sense I will proceed and expand it with other collStats fields

[ishleenk17]

This looks fine. Thanks Stefan.
Let's come up with the most relevant list of metrics that we would like to include for collstats.

[stefans-elastic]

I suggest:

• size
• count
• avgObjSize
• storageSize
• totalIndexSize
• totalSize

Please let me know what do you think.

P.S. I've updated the PoC a bit (introduced WaitGroup's). I think it sould make the performance better.

[ishleenk17]

The list looks good. Lets also add below two to these

max
nindexes

[stefans-elastic]

@ishleenk17 done and PR is ready for review - #42171