Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add storage extension to save filelog receivers' offsets #1718

Merged
merged 5 commits into from
Oct 11, 2023

Conversation

jaehnri
Copy link
Contributor

@jaehnri jaehnri commented Sep 20, 2023

Fixes #1708

The filestorage extension allows us to keep track of the tail offset for every log file watched by the filelog receivers. This way, whenever an opni-collector-agent pod restarts, they can continue reading the log files from where they stopped.

Previously, whenever a crash or restart happened, agents would read from the start again, sending repeated logs and causing an overflow in the aggregator queues:

2023-09-20T14:18:42.911Z	error	exporterhelper/queued_retry.go:216	Dropping data because sending_queue is full. Try increasing queue_size.	{"kind": "exporter", "data_type": "logs", "name": "otlp", "dropped_items": 100}

Now, on restarts, this is logged to the opni-collector-agents:

2023-09-20T15:42:14.081Z	info	extensions/extensions.go:31	Starting extensions...
2023-09-20T15:42:14.081Z	info	extensions/extensions.go:34	Extension is starting...	{"kind": "extension", "name": "file_storage"}
2023-09-20T15:42:14.082Z	info	extensions/extensions.go:38	Extension started.	{"kind": "extension", "name": "file_storage"}
2023-09-20T15:42:14.083Z	info	adapter/receiver.go:45	Starting stanza receiver	{"kind": "receiver", "name": "filelog/k8s", "data_type": "logs"}
2023-09-20T15:42:14.084Z	info	fileconsumer/file.go:354	Resuming from previously known offset(s). 'start_at' setting is not applicable.	{"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "component": "fileconsumer"}

Also, I disabled the audit logs by default as rancher-logging does.

@jaehnri jaehnri force-pushed the store-filelogreceivers-offsets branch 4 times, most recently from 9694669 to ffd9cf6 Compare September 26, 2023 14:18
@jaehnri jaehnri marked this pull request as ready for review September 26, 2023 15:11
Signed-off-by: Joao Henri <joao.henri@suse.com>
Signed-off-by: Joao Henri <joao.henri@suse.com>
Signed-off-by: Joao Henri <joao.henri@suse.com>
Signed-off-by: Joao Henri <joao.henri@suse.com>
Signed-off-by: Joao Henri <joao.henri@suse.com>
@jaehnri jaehnri merged commit e305dab into main Oct 11, 2023
2 checks passed
@jaehnri jaehnri deleted the store-filelogreceivers-offsets branch October 11, 2023 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Save collector-agent log file offset in the host
2 participants