Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda using this handler is forwarding logs from Cloudwatch multiple times #54

Open
nnixen opened this issue Aug 16, 2021 · 2 comments

Comments

@nnixen
Copy link

nnixen commented Aug 16, 2021

Hi,
I'm writing AWS Lambda to forward logs from Cloudwatch to Splunk Cloud. Everything works, but when there is small amount of events (logs pushed to Cloudwatch loggroup) my lambda forwards them multiple times, sometimes 2, sometimes more. I'm positive I don't call this handler multiple times for the same event, so can it be something in the handler itself? I changed this bit of code to get parameters' substitution working like I needed it to do (function format_record()):

temp = json.loads(str(record.getMessage()))
        params = {
            'time': temp['_time'],
            'host': temp['_host'],
            'index': temp['_index'],
            'source': temp['_source'],
            'sourcetype': temp['_sourcetype'],
            'event': temp['event']
        }

Is it possible that it broke something and now handler workers are forwarding same logs multiple times? I make list of all Cloudwatch events and then iterate over it calling this handler:

for message in messages:
                logger.critical(message) 
                splunk.force_flush()
@JeffreyLMelvin
Copy link
Collaborator

If you are going to force_flush every message, why not just set the flush_interval to zero? This will disable the use of a queue and timer and send the logs individually to Splunk

@JeffreyLMelvin
Copy link
Collaborator

WRT the observed duplicate logs, this is also something I have observed, but is not something that I have necessarily attributed to this library. If you are able to identify the cause and can update this issue or open a new one with the details or a PR with a fix, that would be awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants