Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to disable logging to stdout/stderr? #244

Open
dagardner-nv opened this issue Dec 13, 2023 · 1 comment
Open

Is it possible to disable logging to stdout/stderr? #244

dagardner-nv opened this issue Dec 13, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@dagardner-nv
Copy link

Is your feature request related to a problem? Please describe.
Currently camouflage logs to both camouflage.log and stdout/stderror.

Describe the solution you'd like
Not sure if this is already possible, but I would like to configure these separately. Ideally stderr would only contain logs at the error level, while camouflage.log would contain info and above.

Additional context
Currently I launch camouflage as a sub-process from Python. Since camouflage is left running during the duration of the tests, I redirect stdout/stderr to /dev/null to avoid the possibility of the subprocess PIPE from filling up and blocking.

popen = subprocess.Popen(["camouflage", "--config", "config.yml"],
                            cwd=root_dir,
                            stderr=subprocess.DEVNULL,
                            stdout=subprocess.DEVNULL,
                            preexec_fn=_set_pdeathsig(signal.SIGTERM))

While this works, and I can print the contents of camouflage.log in the event of an error, this fails to cover the cases where camouflage is unable to launch such as not being installed, or the system having some other issue.

@dagardner-nv dagardner-nv added the enhancement New feature or request label Dec 13, 2023
@dagardner-nv
Copy link
Author

I poked around, it appears the console transport is always present:
src/logger/index.ts:

  transports: [
    new winston.transports.Console(),
    new winston.transports.File({
      filename: path.join(process.cwd(), "camouflage.log"),
    }),

dagardner-nv added a commit to dagardner-nv/Morpheus that referenced this issue Dec 14, 2023
Improve camouflage error reporting by redirecting stdout and stderr to a file.
This is a work-around for not being able to control the log level of camouflage's console logger
testinggospels/camouflage#244

We can't redirect stdout/stderr to a PIPE since on successful startup the PIPE would fill and block
camouflage. On the other hand sending stdout/stderr to /dev/null prevents users from seeing errors
from the shell/OS regarding camouflage failing to start.
rapids-bot bot pushed a commit to nv-morpheus/Morpheus that referenced this issue Dec 20, 2023
This fixes three issues:

1. Camouflage not starting up within the 10s timeout when the CI system is under high load. This was fixed by simply increasing the timeout to 30s.
2. Camouflage startup errors (command not fount, out of memory etc...) not being reported since stdout/stderr were being sent to `/dev/null`. This was fixed by redirecting stdout/stderr to a file, and displaying the content on failure.

This is a work-around for not being able to control the log level of camouflage's console logger: 
testinggospels/camouflage#244

We can't redirect stdout/stderr to a PIPE since on successful startup the PIPE would fill and block camouflage.

Closes #1417 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1436
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants