Fixed deadlock when rolling the file under a heavy load

[dmatej]

• Fixed LogRecordBuffer configuration - it always used defaults
  • That caused overload without any possibility to configure maxWaitTime or increase the buffer size
• Fixed deadlock - don't use loggers in critical section.
  • Deadlock can be caused when we are rolling the file with the full log record buffer AND we log anything from this thread.
  • Solution: if you really need to log something, use asynchronous thread.
  • Under a heavy load (1 record every 3 microseconds) such log record can occur even after another 5000 other records in the buffer or trying to be added to the buffer. It is just another one. Usually it will be the first one.
• Moving from synchronized to reentrant locks + fixed deadlock

[avpinchuk]

The old code did not guarantee visibility of variables. Each new thread could see the old values (or mix from old and new values) and do reconfiguration.

This code is now safe, although somewhat slower :)

[avpinchuk]

Hmmm, if we move check under write lock, we have deadlock...

UPD. pollOrWait() method blocked on ABQ.take() with acquired read lock. Then reconfigure() tried to acqure write lock and wait indefinitely.

[dmatej]

Good lesson, I read the javadoc again, and also - I should never push without running tests of the module at least.

[dmatej]

Well, now I passed tests, but I did some manual testing and I have another deadlock. Also I see too many logging pumps for STDOUT+STDERR, seems after every reconfiguration it creates another ...

EDIT: Oh, I got it. Those pollOrWait are waiting forever locked, so we can never get the write lock!

And I forgot also about the possibility to decrease the size of the full buffer.