Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for asynchronous logging #202

Open
DarrenCook opened this issue Aug 27, 2014 · 8 comments
Open

Support for asynchronous logging #202

DarrenCook opened this issue Aug 27, 2014 · 8 comments

Comments

@DarrenCook
Copy link

(I think this is a feature request, but if it already exists, it is a request for a usage example. Thanks!)

There is mention, in 2013, that version 9.0 will support asynchronous logging, but I cannot see anything after that. I was hoping there would be a thread-safe queue, behind the scenes, and a dedicated thread that takes items from the queue and writes them to the disk file. I.e. adding a string to a queue is quick, so doesn't lock a thread for long, but appending to a disk file can be relatively slow.

@abumq
Copy link
Owner

abumq commented Aug 28, 2014

Several attempts have been made but nothing very solid so far
For those wondering about "mention in 2013", it's here

leaving it open for suggestions (and/or comments)

@abumq
Copy link
Owner

abumq commented Aug 28, 2014

Aiming to have an experimental - which will be enabled by _ELPP_EXPERIMENTAL_ASYNC - in early stages at the moment - a lot of issues

@abumq abumq added the accepted label Aug 28, 2014
@abumq
Copy link
Owner

abumq commented Aug 29, 2014

Hi Darren,

thanks for the email but I have already implemented experimental async logging and would love if you could give it a test on your PC - it's in really early stage at the moment and I need to ensure I test across all the platforms to ensure availability of all the headers (and handle different cases) - it currently uses POSIX thread model but obviously it will use native threading model by corresponding platform.

I had it tested on mac and it's not a big success there, i have tested in linux (using Intel C++ Compiler) and it's going good.

In order for you to test it, get a copy of easylogging++.h header from develop branch and compile any program (may be an experimental program in https://github.com/easylogging/easyloggingpp/blob/develop/src/experimental/async/prog.cpp) and define macro _ELPP_EXPERIMENTAL_ASYNC to enable asynchronous logging (use build.sh)

let me know how you go.

Thanks

@abumq abumq changed the title Async queue option (or example) Support for asynchronous logging Aug 29, 2014
@abumq
Copy link
Owner

abumq commented Aug 29, 2014

NOTE: This issue is not to be closed until asynchronous logging is stable (i.e, out of experimental mode)

@abumq
Copy link
Owner

abumq commented Aug 30, 2014

Dead lock in multithreaded app (ran pthread.cpp from samples/STL)

g++ pthread.cpp -o bin/pthread.cpp.bin  -D_ELPP_DEBUG_ERRORS -D_ELPP_THREAD_SAFE -D_ELPP_STL_LOGGING -D_ELPP_LOG_UNORDERED_SET -D_ELPP_LOG_UNORDERED_MAP -D_ELPP_STACKTRACE_ON_CRASH -D_ELPP_LOGGING_FLAGS_FROM_ARG -std=c++11 -pthread -Wall -Wextra -pedantic -pedantic-errors -Werror -Wfatal-errors -D_ELPP_EXPERIMENTAL_ASYNC 
./bin/pthread.cpp.bin

@DarrenCook
Copy link
Author

Can you describe/show how you've implemented it? (The source code is one huge file, littered with conditional compilation lines, so is hard to find things in.)

Just the high-level description is enough (e.g. std::queue protected by a mutex and std::lock_guard for pushing to the queue, then calling condition_variable.notify(). Then a single dedicated thread, using
std::unique_lock to find and remove strings from the queue, and writing them out to file.)

It is so easy to get multi-threading wrong that I try to copy patterns/code from threading experts.

@abumq
Copy link
Owner

abumq commented Aug 30, 2014

Above comment was just reminder for myself - well you are right - this is high level on how it works - right now it's using single queue (not good i know), but since it is in experimental mode so i may keep changing the strategy as i go - but for now yes you are right in terms of how it works but it's just another Async callback that I added instead of changing whole way of how logging work I had it implemented so it's extendable. (if you interested here is line where it happens)

Any way the strategy that i may change it to is every logger may (or may not have - depending on file it's pushing to) it's own dispatch worker. This will fasten up the process since i have seen great improvement with async logging yet clean bit (cleaning queue when application wants to close) takes longer than expected because one thread is busy in cleaning (and performing I/O) massive logs - checkout src/experimental/async/prog.cpp for example - it says "finished" sooner than synchronous logging but cleaning takes longer than expected. another thing is starting worker waits for 5 sec before it start work because of Storage initialization (construction) - need to look at all these conditions and improve code readability; So need to improve this - can't live with it in stable releases. Also dead lock will be looked at when i get chance to work on the project as i normally do it more often over weekends (because of full time work)

I try never to copy code but to understand it and then write my own version so that i understand it inside out :)

@baiwfg2
Copy link

baiwfg2 commented Oct 20, 2017

@abumusamq Hi. I'm wondering how the async logging functionality going.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants