cherrypick-1.1: util/log: don't panic

[benesch]

Cherry-pick of #17871. Fixes, most likely, #19202.

cc @cockroachdb/release

---

Previously, log.outputLogEntry could panic while holding the log mutex.
This would deadlock any goroutine that logged while recovering from the
panic, which is approximately all of the recover routines. Most
annoyingly, the crash reporter would deadlock, swallowing the cause of
the panic.

Avoid panicking while holding the log mutex and use l.exit instead,
which exists for this very purpose. In the process, enforce the
invariant that l.mu is held when l.exit is called. (The previous
behavior was, in fact, incorrect, as l.flushAll should not be called
without holding l.mu.)

Also add a Tcl test to ensure this doesn't break in the future.

[cockroach-teamcity]

This change is 

[benesch]

Needed to apply the following patch to get tests to pass:

diff --git a/pkg/cli/interactive_tests/test_missing_log_output.tcl b/pkg/cli/interactive_tests/test_missing_log_output.tcl
index 58e219737..f1c391c85 100644
--- a/pkg/cli/interactive_tests/test_missing_log_output.tcl
+++ b/pkg/cli/interactive_tests/test_missing_log_output.tcl
@@ -27,7 +27,7 @@ system "grep -F 'log: exiting because of error: write /dev/stderr: broken pipe'
 end_test
 
 start_test "Check that a broken log file prints a message to stderr."
-send "$argv start -s=path=/dev/null --insecure --logtostderr\r"
+send "$argv start -s=path=logs/db --log-dir=/dev/null --insecure --logtostderr\r"
 eexpect "log: exiting because of error: log: cannot create log: open"
 eexpect "not a directory"
 eexpect ":/# "

This is a more sensible test, anyway. I'll send that patch on master.

[benesch]

Never mind; @andreimatei already made the equivalent fix on master.