Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flush streams #32

Closed
m-ou-se opened this issue May 23, 2020 · 6 comments
Closed

Flush streams #32

m-ou-se opened this issue May 23, 2020 · 6 comments

Comments

@m-ou-se
Copy link
Owner

m-ou-se commented May 23, 2020

Python uses the stdout and stderr FILE*s from libc, which by default only flush on \n when the output is a tty.

This means that when the output is not a tty, output can be lost when Rust code doesn't flush the C streams before exiting, such as in a panic.

This explains why no error message was seen in #31 at first.

@m-ou-se
Copy link
Owner Author

m-ou-se commented May 23, 2020

Unfortunately, the libc crate doesn't expose the stdout and stderr FILE*s. :(

@m-ou-se m-ou-se changed the title Flush C streams Flush streams May 23, 2020
@m-ou-se
Copy link
Owner Author

m-ou-se commented May 23, 2020

Just flushing stdout/stderr using fflush() doesn't seem to work. Seems like CPython does its own buffering.

Adding sys.stdout.flush() in Python does work.

@m-ou-se
Copy link
Owner Author

m-ou-se commented May 23, 2020

Looks like CPython calls sys.stdout.flush() and sys.stderrr.flush() internally during cleanup: https://github.com/python/cpython/blob/1cba1c9abadf76f458ecf883a48515aa3b534dbd/Python/pylifecycle.c#L1204-L1232

But it doesn't seem to expose a function that does that doesn't exit/cleanup everything.

@m-ou-se
Copy link
Owner Author

m-ou-se commented May 23, 2020

PyO3 also tried to work around this problem, by simply executing import sys; sys.stderr.flush() before panicking:

https://github.com/PyO3/pyo3/blob/072be6ce8376edae60963bf0e6db7c32a28e144d/src/lib.rs#L304-L305

@m-ou-se
Copy link
Owner Author

m-ou-se commented May 23, 2020

Seems like the right thing to do is to call Py_Finalize before exiting. That'll flush the streams, join threads, etc. But running that function with atexit() results in a segfault.

@m-ou-se
Copy link
Owner Author

m-ou-se commented May 23, 2020

Segfault doesn't occur when acquiring the GIL first :)

Opened a PR on PyO3: PyO3/pyo3#943

@m-ou-se m-ou-se closed this as completed Jul 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant