-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HowTo: Telemetry Logging to File when running as windows service #5300
Comments
Still having the same problems in v0.66.0 |
As far as I know, we don't have maintainers using Windows. This is unlikely to move forward without help from people running the collector on Windows. |
Still an issue in v0.92.0:
Really wish the collector could have see some Windows love 💔 |
As far as I know, we are still in the same position: people who care about OTel Collector on Windows should join us and help move this forward. |
I'd help if I could, but I've never used Go before, and wouldn't even know how to start with something like debugging why logs aren't being written 🤷. Maybe I'll get some time to tinker one of these days... |
Hi @cocowalla - I do work with both golang and Windows and can take a look at this, but, would appreciate someone double-checking the configurations and extra testing. You should be able to get rid of the @jpkrohling you can assign this one to me. |
@pjanotti Great! I did actually have a go the other day, and got it to build easily enough, but it's the "contrib" version I need and could never get changes I made in the "main" version to apply in the "contrib" version when I built it. 🤷♂️ Setting up an event source did indeed get rid of the |
Setting the
This redirection to the Event Log is done much earlier than the code loading the configuration and setting up the telemetry so to consider disabling the Event Log requires us to be a bit careful. For quite some time the default behavior of the collector is to log to the Event Log when running on Windows as service and although there are a few issues complaining about that I think many users are fine (and quiet) with that default. At first it seems that the best option is to add a new telemetry configuration option to control the log levels sent to the Event Log (so one could avoid the more verbose messages) and a way to disable it completely. This way we preserve the current behavior but allow the ones that want to turn down/off the usage of Event Log on Windows. @cocowalla the changes should be done only on "core" (the current repo), the contrib version will be updated after a core release with the changes. Before moving ahead with a change proposal I'm planning to review the issues related to usage of the Event Log by the collector. |
) **Description:** Fixes #5300 With this change the service telemetry section is respected by the collector when running as a Windows service. Log lever can be used to control the verbosity of the events logged and the logger can be redirected to a file by specifying an output path on the service telemetry config. By default `stdout` and `stderr` are redirected to the event log when running as a Windows service to keep the current behavior. The code change itself was made with a focus of not breaking the public APIs and not reading the config more than once. That said it is probably something to be refactored when the public APIs can be touched again. **Link to tracking Issue:** #5300 **Testing:** The test is an integration test that depends on the actual executable. It checks for event publication and file output.
Describe the bug
This is no traditional bug report, more a documentation how to solve the described issue.
If maintainers would like a PR to expand the docs anywhere to contain information about this (not sure what from it), I could try to contribute.
If not, this could also considered a rant and be closed. Maybe it helps somebody google landing here.
I spent several HOURS to get a solution for something that should by trivial to do.
The Situation:
otelcol.exe
on windows as a windows-service (supported natively by the binary)exporter
to send them off directly? (e.g. usingloki
)The Goal:
otelcol
to files (to tail and ship to central logging server)json
as log formatSCREENSHOTS: Buggy windows application logs from otelcol (CLICK ME)
this:
![grafik](https://user-images.githubusercontent.com/33002073/166070856-6dfecbd6-47fa-4bd0-a7eb-f7b78dd0bb73.png)
![grafik](https://user-images.githubusercontent.com/33002073/166070885-d623a0a9-30e5-43f7-9c7b-3ee044a89e28.png)
is actually:
and this:
![grafik](https://user-images.githubusercontent.com/33002073/166070907-f08771b3-e9fe-4727-b0a4-a6f50806e52e.png)
![grafik](https://user-images.githubusercontent.com/33002073/166070924-137d16e3-e352-4f5c-b2d2-0b6c1862f19c.png)
is actually:
The journey:
The docs here https://opentelemetry.io/docs/collector/configuration/#service describe the
telemetry
logs and link to here: https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.goWindows Eventlog is not mentioned however.
encoding: json
worked when testing in console. Butstdout
/stderr
are not available in a windows-service and can't be redirected either (afaik). So tried using a filepath:Realize using relatives paths is not good, because windows services use
C:\Windows\System32
(doh) so your logs end up there.Since our installation directory is fixed, I then tried using an absolute file path for
output_paths
, which was unsuccessful.After googling found this: uber-go/zap#621
For nearly 4 yours, logging to absolute file paths on windows is broken in zap ¯\(ツ)/¯
I nearly managed a hack-around by using this path to get out from
system32
:output_paths: ["../../Program Files/OtelCollector/otelcol.log.json"]
which created the logfile in the correct directory, but did not write any logs to it. Could not figure out why.
Also, otelcol source does not mention log filesize limit / log rotation (?), so unsuitable anyway.
Workaround service wrapper
Logging natively seemed not possible. So try using
stdout
and redirect that to a file using 3rd party.https://nssm.cc/ is no longer maintained and dubious, so tried using https://github.com/winsw/winsw
When starting otelcol as a child of winsw as service, it crashed with the following message:
After searching the github repo find this:
The final puzzle peace. Using an env variable in winsw config kept otelcol from crashing.
Logs from stdout where picked up by winsw and logged to a file, enabling filerotation through winsw.
(the files are then tailed by Grafana promtail and sent to loki, since logging in OTEL is still unstable)
What did you expect to see?
An easier way to log otelcol log-messages to a file.
What version did you use?
Version: (e.g.,
v0.48.0
)Environment
OS: Windows 10 / Windows Server 2019
The text was updated successfully, but these errors were encountered: