How to combine multiple request to 1 JSON file?

[dirablue]

@perriea
Hi Thank you for making nice plugin.

I tried and I have a question.

I tested some request with this plugin, and JSON files are created each request.
How can I combine multiple request to 1 JSON file using time rotation, file-size ?

[perriea]

Hi @dirablue this functionality isn't available now. Have you an example (documentation) in another plugin ?

[dirablue]

@perriea
Hi Perriea, thank you for your message.

if this plugin is able to support like rotate_wait, it would be great to me.
https://docs.fluentd.org/input/tail#rotate_wait

I have a plan to send many requests to GCS with this plugin. ( like log data )
But files are separated each request, and GCS fee should be high.

Could you consider to implement of combine data to 1 JSON with rotation option .. ?

[perriea]

Hmm, you can't write all logs in one file. All object unchanging.
But I can wait for the size or the duration (second, minute ...) before sending logs to GCS, example : https://github.com/chestercheng/fluent-bit-go-azblob/blob/devel/cmd/out_azblob/config.go#L104

[dirablue]

@perriea

But I can wait for the size or the duration (second, minute ...) before sending logs to GCS, example :

in this case, some requests will be saved in 1 JSON file? ( not all logs, only some )
it's great if it's possible !

[perriea]

Yes it's not complicated, I see that this we 🚀

[dirablue]

@perriea
wao, thank you so much !
I'm looking forward to use it !

[dirablue]

@perriea
Hi perriea, may I ask you about when you are able to implement it?