Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random key repeating when under high load #545

Closed
neilio opened this issue Jan 9, 2017 · 99 comments
Closed

Random key repeating when under high load #545

neilio opened this issue Jan 9, 2017 · 99 comments
Labels

Comments

@neilio
Copy link

neilio commented Jan 9, 2017

I'm still trying to narrow down if this is definitely Karabiner or something else, but I've noticed if my machine is under high CPU load that typed character often get "stuck" - pressing a key makes it repeat multiple times even though I'm not holding it down.

Has anyone else seen this?

@zhenchaoli
Copy link

Similar issue here. Not 100% sure if it's Karabiner either. But it's one more sample :)

@jessejanderson
Copy link

Same issue here, I've had it happen twice now (once with left arrow, once with volume-down -- both while machine was cranking, fans spinning, etc).

@codyrobbins
Copy link

Same thing here. It’s happening quite often for me on an older 2010 MacBook Air.

@modkaffes
Copy link

modkaffes commented Jan 17, 2017

Same here and it's happening really often, even under light CPU usage.

When using Chrome and ctrl-tabbing to next tab, it seems that the ctrl-tab combination stays "pressed" for a while and it jumps five tabs ahead instead of one. I've also noticed when text editing, the space key gets "stuck" at times.

Only modification I've enabled is non_us_backslash to grave_accent_and_tilde.

Karabiner-Elements version: 0.90.79

@tekezo
Copy link
Member

tekezo commented Jan 17, 2017

Karabiner-Elements leaves the key repeat handling to macOS since v0.90.69.
So, if this issue is happen in the recent version, it's a macOS issue.

@codyrobbins
Copy link

@tekezo I can’t test definitively whether it is Karabiner, because my keyboard still stops working entirely when I quit Karabiner, but I think it may be Karabiner rather than macOS because I don’t remember it starting to occur after a macOS update.

@tisba
Copy link

tisba commented Jan 20, 2017

This might be related to #510, #466 and several other issues.

@tisba tisba mentioned this issue Jan 20, 2017
@chellberg
Copy link

I'm experiencing this too, and I think it started after either the most recent or the second most recent Karabiner update. I don't think I've installed any macOS updates recently.

@Mauin
Copy link

Mauin commented Feb 2, 2017

I'm seeing the same. Using my 60% (GON NerD) standing on top of my MacBook keyboard. Using Karabiner to disable the internal MacBook keyboard. Under high load it seems like turning off the internal keyboard seems to cut out as some keypresses from the internal keyboard get registered and at some point I get the stuck key issue. Only restarting the MacBook solves the stuck key issue for me.

@RobertARandolph
Copy link

This happens to me frequently and turning off karabiner-elements makes the issue go away.

@EleanorRagone
Copy link

I'm having this issue and it is 100% due to karabiner-elements. I absolutely love being able to tweak keyboard to exactly what I want, but if it's going to put in a bunch of random letters at random times, I can't use it! Is there any work on a fix?

@jessejanderson
Copy link

It's pretty bad - consistently seeing this behavior several times a day.

Turning off karabiner-elements fixes the issue.

@tadaszelvys
Copy link

I have this issue too, mbp 2015, on heavier load it becomes a real problem as pressing a key will result in random amount of repeats.

@EleanorRagone
Copy link

So... any updates on this? @tekezo? Anything we can do to help out?

@EleanorRagone
Copy link

Any chance we can get an update on this of some kind? It's getting really, really frustrating to have all my programs quit because karabiner keeps repeating key commands like "cmd-q"

@daniula
Copy link

daniula commented Feb 28, 2017

I have the same issue on new Macbook 13" Early 2015. I'm suspecting Karabiner and since I've disabled it, issue is gone.

Karabiner version: 0.90.86
macOS: 10.12.3

@MartinNuc
Copy link

Had this issue all time time since using Karabiner Elements. Began to use it very early when the only way to config was through editting config file.

Same as others. After I quit Karabiner issue is gone.

Currently Karabiner version: 0.90.86
macOS: 10.12.3

@sprout42
Copy link

sprout42 commented Mar 1, 2017

I've had this happen to me a few times now. Occasionally I get a lag when typing, and worst sometimes delete seems to get stuck and keeps deleting things until I hit the control key (for some reason). I been unable to identify any pattern behind this happening.

Does not happen when Karabiner Elements is not running.

Keyboard: CMStorm (no additional software installed)
Currently Karabiner version: 0.90.86
macOS: 10.12.3

@markschwarz
Copy link

markschwarz commented Mar 2, 2017

I'm uninstalling, too, until this thread wraps up. I'm seeing this behavior roughly 2x week, most often after coming back from sleep (which might just be a type of high load condition). If there's logging I can enable to rule out Karabiner, I'll try it!

I love using my Caps key for vim escape. I'll miss this.

This SO overflow seems loosely related: https://apple.stackexchange.com/questions/13708/disabling-the-keep-pressed-function-of-os-x-sticky-keys

@codyrobbins
Copy link

codyrobbins commented Mar 2, 2017

The issue is definitely related to CPU load—on a newer machine it rarely occurs but on an older 2010 MacBook Air it happens almost constantly when the CPU has a load and it makes Karabiner unusable.

@EleanorRagone
Copy link

@tekezo any update here? Anything we can do to help out? This has become completely unusable for me =/

@SConaway
Copy link

Same here MBP 15" Mid '14

@jessejanderson
Copy link

jessejanderson commented Mar 16, 2017

This is really becoming unusable. I'm completely reliant on having my Caps = Ctrl/Esc otherwise I would uninstall this. Currently, I just have come to expect 1-2 catastrophic key repeats every day now where a sticky key will do something like CMD-Q every single app I have open.

Any sort of update would be appreciated.

@codyrobbins
Copy link

Is a full list of previous releases available somewhere? I was going to try reinstalling previous releases to try to narrow down the release that introduced this problem, in the hope that it would help make the fix more obvious for @tekezo, but I don’t see any way to get the previous releases easily unfortunately.

@daniula
Copy link

daniula commented Mar 17, 2017

@jessejanderson You can remap Caps to Ctrl/Esc without Karabiner. This dialog opens up after clickin "Modifier Keys…"
screen shot 2017-03-17 at 21 03 54

I'm using Karabiner only to remap £ with ~ on my EU keyboard. Not a big deal, so I keep it uninstalled until it's fixed.

@jessejanderson
Copy link

@daniula thanks but I actually need Karabiner to map Caps to Ctrl AND Esc.
Tapping the key = Esc, holding the key as a modifier = Ctrl.

@oleander
Copy link

Any news on this one? Having the same problem :/

@jerzygangi
Copy link

Also having this problem.

@markschwarz
Copy link

I'd totally overlooked the Mac Keyboard menu shortcut, @daniula. That does everything I need: Caps->Esc. Thanks!

@tekezo
Copy link
Member

tekezo commented May 24, 2018

@saagarjha
Thank you for feedback!
It's surely my mistake. I'll separate the blocking I/O from main thread soon.

@saagarjha
Copy link
Contributor

saagarjha commented May 24, 2018

I appreciate the rapid response, as well as your initiative to get this issue resolved! To go into more detail about what needs to be done regarding blocking the main thread, I'd suggest writing your code with realtime constraints in mind. If you haven't done this before, I'd suggest doing some research on how this is done; often you can find some high-quality articles that focus on realtime audio but should apply here as well. In short, you should not be making any blocking calls on the main thread–so disk I/O, mutexes, etc. are certainly off the table. For audio things like allocating memory (e.g. malloc, new) or worst-case complexity (std::vector::push_back) must also be considered; I'm not sure if this is necessary for Karabiner yet. Certainly, nonessential blocking tasks like modifying the caps lock LED or writing out configuration should be delegated to a different thread at the very minimum.

@tekezo
Copy link
Member

tekezo commented May 25, 2018

I changed the I/O handling at v12.0.11.

Please confirm the latest beta version.
Check for beta updates

@saagarjha
Copy link
Contributor

saagarjha commented May 25, 2018

Unfortunately, it seems that this is still an issue. Here's a new sample:

Call graph:
    9688 Thread_27005340   DispatchQueue_1: com.apple.main-thread  (serial)
    + 9688 start  (in libdyld.dylib) + 1  [0x7fff5a107015]
    +   9688 main  (in karabiner_grabber) + 3531  [0x10dc9a58b]
    +     9688 CFRunLoopRun  (in CoreFoundation) + 99  [0x7fff32342c33]
    +       9688 CFRunLoopRunSpecific  (in CoreFoundation) + 483  [0x7fff323041a3]
    +         9688 __CFRunLoopRun  (in CoreFoundation) + 2586  [0x7fff32304e4a]
    +           9688 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__  (in CoreFoundation) + 9  [0x7fff32342c69]
    +             9688 _dispatch_main_queue_callback_4CF  (in libdispatch.dylib) + 776  [0x7fff5a0d9221]
    +               9688 _dispatch_source_invoke  (in libdispatch.dylib) + 620  [0x7fff5a0d0081]
    +                 9688 _dispatch_continuation_pop  (in libdispatch.dylib) + 472  [0x7fff5a0e0e81]
    +                   9688 _dispatch_client_callout  (in libdispatch.dylib) + 8  [0x7fff5a0cddb8]
    +                     9688 invocation function for block in krbn::connection_manager::connection_manager(krbn::version_monitor&, krbn::device_grabber&)  (in karabiner_grabber) + 35  [0x10dcbfcc3]
    +                       9688 SCDynamicStoreCopyConsoleUser  (in SystemConfiguration) + 100  [0x7fff3e886faf]
    +                         9688 SCDynamicStoreCopyValue  (in SystemConfiguration) + 158  [0x7fff3e868f7e]
    +                           9688 configget  (in SystemConfiguration) + 170  [0x7fff3e869a5c]
    +                             9688 mach_msg  (in libsystem_kernel.dylib) + 60  [0x7fff5a24d724]
    +                               9688 mach_msg_trap  (in libsystem_kernel.dylib) + 10  [0x7fff5a24e20a]
    9688 Thread_27005342
    + 9688 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebf9]
    +   9688 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff5a41f50d]
    +     9688 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff5a41f661]
    +       9688 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (spdlog::details::async_log_helper::*)(), spdlog::details::async_log_helper*> >(void*)  (in karabiner_grabber) + 62  [0x10dcb4e9e]
    +         9688 spdlog::details::async_log_helper::worker_loop()  (in karabiner_grabber) + 88  [0x10dcb4108]
    +           9688 spdlog::details::async_log_helper::process_next_msg()  (in karabiner_grabber) + 102  [0x10dcb4556]
    +             9688 spdlog::details::mpmc_bounded_queue<spdlog::details::async_log_helper::async_msg>::dequeue_for(spdlog::details::async_log_helper::async_msg&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)  (in karabiner_grabber) + 446  [0x10dcb4c8e]
    +               9688 std::__1::cv_status std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > > const&)  (in karabiner_grabber) + 117  [0x10dcb4d65]
    +                 9688 std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)  (in libc++.1.dylib) + 93  [0x7fff5805cd43]
    +                   9688 _pthread_cond_wait  (in libsystem_pthread.dylib) + 732  [0x7fff5a420589]
    +                     9688 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff5a257a1e]
    9688 Thread_27005343
    + 9688 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebf9]
    +   9688 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff5a41f50d]
    +     9688 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff5a41f661]
    +       9688 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>::*)(), krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>*> >(void*)  (in karabiner_grabber) + 62  [0x10dcb8a1e]
    +         9688 krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>::loop()  (in karabiner_grabber) + 107  [0x10dcb863b]
    +           9688 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)  (in libc++.1.dylib) + 18  [0x7fff5805ccb0]
    +             9688 _pthread_cond_wait  (in libsystem_pthread.dylib) + 732  [0x7fff5a420589]
    +               9688 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff5a257a1e]
    9688 Thread_27005346
    + 9688 start_wqthread  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebe9]
    +   9688 _pthread_wqthread  (in libsystem_pthread.dylib) + 1035  [0x7fff5a41f009]
    +     9688 __workq_kernreturn  (in libsystem_kernel.dylib) + 10  [0x7fff5a258292]
    9688 Thread_27005502
      9688 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebf9]
        9688 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff5a41f50d]
          9688 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff5a41f661]
            9688 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, krbn::receiver::receiver(krbn::device_grabber&)::'lambda'()> >(void*, void*)  (in karabiner_grabber) + 45  [0x10dd3529d]
              9688 krbn::receiver::worker()  (in karabiner_grabber) + 322  [0x10dd35452]
                9688 krbn::local_datagram_server::receive(boost::asio::mutable_buffer const&, boost::posix_time::time_duration, boost::system::error_code&)  (in karabiner_grabber) + 296  [0x10dd36128]
                  9688 boost::asio::io_context::run_one()  (in karabiner_grabber) + 156  [0x10dd368ac]
                    9688 boost::asio::detail::scheduler::run_one(boost::system::error_code&)  (in karabiner_grabber) + 281  [0x10dd37a59]
                      9688 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)  (in karabiner_grabber) + 337  [0x10dd37d51]
                        9688 boost::asio::detail::kqueue_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&)  (in karabiner_grabber) + 273  [0x10dd38181]
                          9688 kevent  (in libsystem_kernel.dylib) + 10  [0x7fff5a258bf2]

Seems like SCDynamicStoreCopyConsoleUser is a blocking method? Here's what I did to reliably trigger and diagnose the issue, if you're curious:

$ echo "int main() { while (1) {} }" | clang -x c -o while -
$ sudo echo # So that the next sudo doesn't ask for a password
$ for i in `seq 1 100`; do ./while &disown ; done; sleep 5; sudo sample `pgrep karabiner_grabber` > sample.txt; killall while

Doing the above with Karabiner not running works as expected, with no key repeats.

@ignaciovazquez
Copy link

ignaciovazquez commented May 25, 2018

I can confirm @saagarjha sampling method also triggers the issue in my machine.

Here's the stack.

OS Version:      Mac OS X 10.13.1 (17B1003)
Report Version:  7
Analysis Tool:   /usr/bin//sample
----

Call graph:
    9742 Thread_46446930   DispatchQueue_1: com.apple.main-thread  (serial)
    + 9742 start  (in libdyld.dylib) + 1  [0x7fff54a67145]
    +   9742 main  (in karabiner_grabber) + 3531  [0x106a7958b]
    +     9742 CFRunLoopRun  (in CoreFoundation) + 99  [0x7fff2d4a9993]
    +       9742 CFRunLoopRunSpecific  (in CoreFoundation) + 483  [0x7fff2d46afa3]
    +         6101 __CFRunLoopRun  (in CoreFoundation) + 1783  [0x7fff2d46b927]
    +         ! 6101 __CFRunLoopServiceMachPort  (in CoreFoundation) + 341  [0x7fff2d46c5d5]
    +         !   6101 mach_msg  (in libsystem_kernel.dylib) + 60  [0x7fff54bad390]
    +         !     6101 mach_msg_trap  (in libsystem_kernel.dylib) + 10  [0x7fff54bade76]
    +         1911 __CFRunLoopRun  (in CoreFoundation) + 2848  [0x7fff2d46bd50]
    +         ! 1911 __CFRunLoopDoSource1  (in CoreFoundation) + 533  [0x7fff2d474165]
    +         !   1911 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__  (in CoreFoundation) + 41  [0x7fff2d474219]
    +         !     1911 __CFMachPortPerform  (in CoreFoundation) + 347  [0x7fff2d47438b]
    +         !       1911 __IOHIDQueueValueAvailableCallback  (in IOKit) + 64  [0x7fff2fc143e4]
    +         !         1911 CFDictionaryApplyFunction  (in CoreFoundation) + 192  [0x7fff2d426250]
    +         !           1911 CFBasicHashApply  (in CoreFoundation) + 106  [0x7fff2d42630a]
    +         !             1911 __CFDictionaryApplyFunction_block_invoke  (in CoreFoundation) + 22  [0x7fff2d426346]
    +         !               1449 krbn::human_interface_device::queue_value_available_callback()  (in karabiner_grabber) + 2764  [0x106b30d4c]
    +         !               : 1449 krbn::device_grabber::value_callback(krbn::human_interface_device&, krbn::event_queue&)  (in karabiner_grabber) + 68  [0x106b2be54]
    +         !               :   1329 boost::signals2::detail::signal_impl<void (), boost::signals2::optional_last_value<void>, int, std::__1::less<int>, boost::function<void ()>, boost::function<void (boost::signals2::connection const&)>, boost::signals2::mutex>::operator()()  (in karabiner_grabber) + 616  [0x106ac6c38]
    +         !               :   | 1329 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type>, std::__1::__list_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::__1::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body<std::__1::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> >::dereference() const  (in karabiner_grabber) + 60  [0x106ac779c]
    +         !               :   |   1093 krbn::device_grabber::manipulate(unsigned long long)  (in karabiner_grabber) + 115  [0x106b255f3]
    +         !               :   |   + 1093 krbn::manipulator::details::post_event_to_virtual_devices::queue::post_events(krbn::virtual_hid_device_client&)  (in karabiner_grabber) + 619  [0x106b261db]
    +         !               :   |   +   1029 krbn::virtual_hid_device_client::call_method(std::__1::function<int ()>)  (in karabiner_grabber) + 56  [0x106b0b438]
    +         !               :   |   +   ! 565 IOConnectCallStructMethod  (in IOKit) + 38  [0x7fff2fbd6227]
    +         !               :   |   +   ! : 565 IOConnectCallMethod  (in IOKit) + 186  [0x7fff2fbd50f4]
    +         !               :   |   +   ! :   565 io_connect_method  (in IOKit) + 369  [0x7fff2fbd52c7]
    +         !               :   |   +   ! :     565 mach_msg  (in libsystem_kernel.dylib) + 60  [0x7fff54bad390]
    +         !               :   |   +   ! :       565 mach_msg_trap  (in libsystem_kernel.dylib) + 10  [0x7fff54bade76]
    +         !               :   |   +   ! 464 IOConnectCallStructMethod  (in IOKit) + 42  [0x7fff2fbd622b]
    +         !               :   |   +   64 krbn::virtual_hid_device_client::call_method(std::__1::function<int ()>)  (in karabiner_grabber) + 35  [0x106b0b423]
    +         !               :   |   233 krbn::device_grabber::manipulate(unsigned long long)  (in karabiner_grabber) + 75  [0x106b255cb]
    +         !               :   |   + 233 krbn::manipulator::manipulator_managers_connector::connection::manipulate(unsigned long long)  (in karabiner_grabber) + 137  [0x106b257d9]
    +         !               :   |   +   131 krbn::manipulator::manipulator_manager::manipulate(std::__1::shared_ptr<krbn::event_queue> const&, std::__1::shared_ptr<krbn::event_queue> const&, unsigned long long)  (in karabiner_grabber) + 1399  [0x106b25e17]
    +         !               :   |   +   ! 131 krbn::manipulator::details::basic::manipulate(krbn::event_queue::queued_event&, krbn::event_queue const&, std::__1::shared_ptr<krbn::event_queue> const&, unsigned long long)  (in karabiner_grabber) + 507  [0x106ad1e0b]
    +         !               :   |   +   !   131 krbn::manipulator::details::basic::from_event_definition::test_event(krbn::event_queue::queued_event::event const&, krbn::manipulator::details::event_definition const&)  (in karabiner_grabber) + 24  [0x106ada098]
    +         !               :   |   +   !     131 krbn::event_queue::queued_event::event::get_key_code() const  (in karabiner_grabber) + 59  [0x106ac15fb]
    +         !               :   |   +   !       131 boost::detail::variant::invoke_visitor<boost::detail::variant::get_visitor<krbn::key_code const>, false>::result_type boost::detail::variant::visitation_impl<mpl_::int_<0>, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<12l>, krbn::key_code, boost::mpl::l_item<mpl_::long_<11l>, krbn::consumer_key_code, boost::mpl::l_item<mpl_::long_<10l>, krbn::pointing_button, boost::mpl::l_item<mpl_::long_<9l>, krbn::pointing_motion, boost::mpl::l_item<mpl_::long_<8l>, long long, boost::mpl::l_item<mpl_::long_<7l>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, boost::mpl::l_item<mpl_::long_<6l>, std::__1::vector<krbn::input_source_selector, std::__1::allocator<krbn::input_source_selector> >, boost::mpl::l_item<mpl_::long_<5l>, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int>, boost::mpl::l_item<mpl_::long_<4l>, krbn::mouse_key, boost::mpl::l_item<mpl_::long_<3l>, krbn::manipulator_environment::frontmost_application, boost::mpl::l_item<mpl_::long_<2l>, krbn::input_source_identifiers, boost::mpl::l_item<mpl_::long_<1l>, boost::blank, boost::mpl::l_end> > > > > > > > > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >, boost::detail::variant::invoke_visitor<boost::detail::variant::get_visitor<krbn::key_code const>, false>, void const*, boost::variant<krbn::key_code, krbn::consumer_key_code, krbn::pointing_button, krbn::pointing_motion, long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::vector<krbn::input_source_selector, std::__1::allocator<krbn::input_source_selector> >, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int>, krbn::mouse_key, krbn::manipulator_environment::frontmost_application, krbn::input_source_identifiers, boost::blank>::has_fallback_type_>(int, int, boost::detail::variant::invoke_visitor<boost::detail::variant::get_visitor<krbn::key_code const>, false>&, void const*, mpl_::bool_<false>, boost::variant<krbn::key_code, krbn::consumer_key_code, krbn::pointing_button, krbn::pointing_motion, long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::vector<krbn::input_source_selector, std::__1::allocator<krbn::input_source_selector> >, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int>, krbn::mouse_key, krbn::manipulator_environment::frontmost_application, krbn::input_source_identifiers, boost::blank>::has_fallback_type_, mpl_::int_<0>*, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<12l>, krbn::key_code, boost::mpl::l_item<mpl_::long_<11l>, krbn::consumer_key_code, boost::mpl::l_item<mpl_::long_<10l>, krbn::pointing_button, boost::mpl::l_item<mpl_::long_<9l>, krbn::pointing_motion, boost::mpl::l_item<mpl_::long_<8l>, long long, boost::mpl::l_item<mpl_::long_<7l>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, boost::mpl::l_item<mpl_::long_<6l>, std::__1::vector<krbn::input_source_selector, std::__1::allocator<krbn::input_source_selector> >, boost::mpl::l_item<mpl_::long_<5l>, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int>, boost::mpl::l_item<mpl_::long_<4l>, krbn::mouse_key, boost::mpl::l_item<mpl_::long_<3l>, krbn::manipulator_environment::frontmost_application, boost::mpl::l_item<mpl_::long_<2l>, krbn::input_source_identifiers, boost::mpl::l_item<mpl_::long_<1l>, boost::blank, boost::mpl::l_end> > > > > > > > > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >*)  (in karabiner_grabber) + 0  [0x106ac3210]
    +         !               :   |   +   101 krbn::manipulator::manipulator_manager::manipulate(std::__1::shared_ptr<krbn::event_queue> const&, std::__1::shared_ptr<krbn::event_queue> const&, unsigned long long)  (in karabiner_grabber) + 1318  [0x106b25dc6]
    +         !               :   |   +   1 krbn::manipulator::manipulator_manager::manipulate(std::__1::shared_ptr<krbn::event_queue> const&, std::__1::shared_ptr<krbn::event_queue> const&, unsigned long long)  (in karabiner_grabber) + 1324  [0x106b25dcc]
    +         !               :   |   +     1 krbn::manipulator::details::basic::already_manipulated(krbn::event_queue::queued_event const&)  (in karabiner_grabber) + 83  [0x106ad1af3]
    +         !               :   |   3 krbn::device_grabber::manipulate(unsigned long long)  (in karabiner_grabber) + 168  [0x106b25628]
    +         !               :   |     3 krbn::manipulator::manipulator_managers_connector::connection::make_input_event_time_stamp_with_input_delay() const  (in karabiner_grabber) + 28  [0x106b295dc]
    +         !               :   |       3 std::__1::__shared_weak_count::lock()  (in libc++.1.dylib) + 8  [0x7fff52ad6974]
    +         !               :   120 boost::signals2::detail::signal_impl<void (), boost::signals2::optional_last_value<void>, int, std::__1::less<int>, boost::function<void ()>, boost::function<void (boost::signals2::connection const&)>, boost::signals2::mutex>::operator()()  (in karabiner_grabber) + 642  [0x106ac6c52]
    +         !               :     120 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type>, std::__1::__list_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::__1::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body<std::__1::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (), boost::function<void ()> >, boost::signals2::mutex> >::lock_next_callable() const  (in karabiner_grabber) + 119  [0x106ac8477]
    +         !               :       120 pthread_mutex_lock  (in libsystem_pthread.dylib) + 0  [0x7fff54cef3d5]
    +         !               193 krbn::human_interface_device::queue_value_available_callback()  (in karabiner_grabber) + 76  [0x106b302cc]
    +         !               : 193 IOHIDQueueCopyNextValueWithTimeout  (in IOKit) + 45  [0x7fff2fc1444e]
    +         !               :   103 IOHIDQueueClass::copyNextEventValue(__IOHIDValue**, unsigned int, unsigned int)  (in IOHIDLib) + 322  [0x106c34512]
    +         !               :   | 103 IOHIDDeviceClass::getElement(unsigned int)  (in IOHIDLib) + 82  [0x106c32146]
    +         !               :   |   103 IOHIDDeviceClass::createElement(__CFData const*, IOHIDElementStruct*, unsigned int, void const*, __CFDictionary*, bool*, unsigned int)  (in IOHIDLib) + 114  [0x106c311ee]
    +         !               :   |     103 CFDictionaryGetValue  (in CoreFoundation) + 131  [0x7fff2d3f1fa3]
    +         !               :   |       103 CFBasicHashFindBucket  (in CoreFoundation) + 3328  [0x7fff2d3f2cf0]
    +         !               :   90 IOHIDQueueClass::_copyNextEventValue(void*, __IOHIDValue**, unsigned int, unsigned int)  (in IOHIDLib) + 15  [0x106c34657]
    +         !               146 krbn::human_interface_device::queue_value_available_callback()  (in karabiner_grabber) + 231  [0x106b30367]
    +         !               : 146 _CFRelease  (in CoreFoundation) + 1056  [0x7fff2d544220]
    +         !               :   146 free_tiny  (in libsystem_malloc.dylib) + 296  [0x7fff54c27108]
    +         !               122 krbn::human_interface_device::queue_value_available_callback()  (in karabiner_grabber) + 256  [0x106b30380]
    +         !               : 122 krbn::event_queue::make_queued_events(std::__1::vector<krbn::hid_value, std::__1::allocator<krbn::hid_value> > const&, krbn::device_id)  (in karabiner_grabber) + 2233  [0x106b316c9]
    +         !               :   122 krbn::event_queue::make_queued_events(std::__1::vector<krbn::hid_value, std::__1::allocator<krbn::hid_value> > const&, krbn::device_id)::'lambda'()::operator()() const  (in karabiner_grabber) + 649  [0x106b31ee9]
    +         !               1 krbn::human_interface_device::queue_value_available_callback()  (in karabiner_grabber) + 2772  [0x106b30d54]
    +         !                 1 krbn::event_queue::clear_events()  (in karabiner_grabber) + 51  [0x106b256d3]
    +         1603 __CFRunLoopRun  (in CoreFoundation) + 2586  [0x7fff2d46bc4a]
    +         ! 1603 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__  (in CoreFoundation) + 9  [0x7fff2d4a99c9]
    +         !   1603 _dispatch_main_queue_callback_4CF  (in libdispatch.dylib) + 776  [0x7fff54a392fd]
    +         !     1603 _dispatch_source_invoke  (in libdispatch.dylib) + 620  [0x7fff54a30154]
    +         !       1603 _dispatch_continuation_pop  (in libdispatch.dylib) + 472  [0x7fff54a40fbc]
    +         !         1603 _dispatch_client_callout  (in libdispatch.dylib) + 8  [0x7fff54a2de88]
    +         !           1223 invocation function for block in krbn::connection_manager::connection_manager(krbn::version_monitor&, krbn::device_grabber&)  (in karabiner_grabber) + 35  [0x106a9ecc3]
    +         !           : 1223 SCDynamicStoreCopyConsoleUser  (in SystemConfiguration) + 100  [0x7fff39749003]
    +         !           :   1223 SCDynamicStoreCopyValue  (in SystemConfiguration) + 158  [0x7fff3972afee]
    +         !           :     1223 configget  (in SystemConfiguration) + 170  [0x7fff3972baec]
    +         !           :       1223 mach_msg  (in libsystem_kernel.dylib) + 60  [0x7fff54bad390]
    +         !           :         1223 mach_msg_trap  (in libsystem_kernel.dylib) + 10  [0x7fff54bade76]
    +         !           380 krbn::device_grabber::update_caps_lock_led(bool)  (in karabiner_grabber) + 219  [0x106ac01fb]
    +         !             380 krbn::core_configuration::profile::get_device_manipulate_caps_lock_led(krbn::device_identifiers const&) const  (in karabiner_grabber) + 322  [0x106ac9b22]
    +         !               251 krbn::core_configuration::profile::device::device(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> const&)  (in karabiner_grabber) + 393  [0x106ab4569]
    +         !               | 126 krbn::core_configuration::profile::simple_modifications::handle_json(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> const&)  (in karabiner_grabber) + 1509  [0x106aa7f45]
    +         !               | + 126 krbn::core_configuration::profile::simple_modifications::erase_by_from_json_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)  (in karabiner_grabber) + 110  [0x106aa8cfe]
    +         !               | +   126 nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::parse(nlohmann::detail::input_adapter, std::__1::function<bool (int, nlohmann::detail::parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >::parse_event_t, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>&)>, bool)  (in karabiner_grabber) + 393  [0x106aa9599]
    +         !               | +     126 nlohmann::detail::parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >::parse(bool, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>&)  (in karabiner_grabber) + 36  [0x106aa9c04]
    +         !               | +       126 nlohmann::detail::lexer<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >::scan()  (in karabiner_grabber) + 40  [0x106aaa7f8]
    +         !               | +         126 nlohmann::detail::lexer<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >::get()  (in karabiner_grabber) + 155  [0x106aaabbb]
    +         !               | +           126 operator new(unsigned long)  (in libc++abi.dylib) + 40  [0x7fff52af5628]
    +         !               | +             126 malloc  (in libsystem_malloc.dylib) + 24  [0x7fff54c0e50b]
    +         !               | +               126 malloc_zone_malloc  (in libsystem_malloc.dylib) + 103  [0x7fff54c0f201]
    +         !               | +                 126 szone_malloc_should_clear  (in libsystem_malloc.dylib) + 82  [0x7fff54c0f2af]
    +         !               | 124 krbn::core_configuration::profile::simple_modifications::handle_json(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> const&)  (in karabiner_grabber) + 949  [0x106aa7d15]
    +         !               | + 124 nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::dump(int, char, bool) const  (in karabiner_grabber) + 349  [0x106a94d8d]
    +         !               | +   124 operator new(unsigned long)  (in libc++abi.dylib) + 40  [0x7fff52af5628]
    +         !               | +     124 malloc  (in libsystem_malloc.dylib) + 24  [0x7fff54c0e50b]
    +         !               | +       124 malloc_zone_malloc  (in libsystem_malloc.dylib) + 103  [0x7fff54c0f201]
    +         !               | +         124 szone_malloc_should_clear  (in libsystem_malloc.dylib) + 422  [0x7fff54c0f403]
    +         !               | +           124 tiny_malloc_from_free_list  (in libsystem_malloc.dylib) + 431  [0x7fff54c10384]
    +         !               | +             124 set_tiny_meta_header_in_use  (in libsystem_malloc.dylib) + 210  [0x7fff54c26f7a]
    +         !               | 1 krbn::core_configuration::profile::simple_modifications::handle_json(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> const&)  (in karabiner_grabber) + 1099  [0x106aa7dab]
    +         !               |   1 nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::dump(int, char, bool) const  (in karabiner_grabber) + 496  [0x106a94e20]
    +         !               |     1 free_tiny  (in libsystem_malloc.dylib) + 628  [0x7fff54c27254]
    +         !               |       1 tiny_free_no_lock  (in libsystem_malloc.dylib) + 1450  [0x7fff54c26ac8]
    +         !               |         1 tiny_free_list_remove_ptr  (in libsystem_malloc.dylib) + 6  [0x7fff54c11647]
    +         !               129 krbn::core_configuration::profile::device::device(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> const&)  (in karabiner_grabber) + 406  [0x106ab4576]
    +         !                 129 nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::json_value::destroy(nlohmann::detail::value_t)  (in karabiner_grabber) + 113  [0x106a9bdf1]
    +         !                   129 nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::json_value::destroy(nlohmann::detail::value_t)  (in karabiner_grabber) + 44  [0x106a9bdac]
    +         !                     129 std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >, std::__1::less<void>, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> > > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >, void*>*)  (in karabiner_grabber) + 54  [0x106a9bb26]
    +         !                       129 nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::json_value::destroy(nlohmann::detail::value_t)  (in karabiner_grabber) + 44  [0x106a9bdac]
    +         !                         129 std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >, std::__1::less<void>, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> > > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> >, void*>*)  (in karabiner_grabber) + 54  [0x106a9bb26]
    +         !                           129 free_tiny  (in libsystem_malloc.dylib) + 628  [0x7fff54c27254]
    +         !                             129 tiny_free_no_lock  (in libsystem_malloc.dylib) + 666  [0x7fff54c267b8]
    +         127 __CFRunLoopRun  (in CoreFoundation) + 1293  [0x7fff2d46b73d]
    +           127 __CFRunLoopDoSources0  (in CoreFoundation) + 127  [0x7fff2d46c26f]
    +             127 CFSetApplyFunction  (in CoreFoundation) + 192  [0x7fff2d439c70]
    +               127 CFBasicHashApply  (in CoreFoundation) + 106  [0x7fff2d42630a]
    +                 127 __CFSetApplyFunction_block_invoke  (in CoreFoundation) + 18  [0x7fff2d439cd2]
    +                   127 __CFRunLoopCollectSources0  (in CoreFoundation) + 21  [0x7fff2d46c3b5]
    9742 Thread_46446935
    + 9742 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff54cf0c5d]
    +   9742 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff54cf156d]
    +     9742 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff54cf16c1]
    +       9742 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (spdlog::details::async_log_helper::*)(), spdlog::details::async_log_helper*> >(void*)  (in karabiner_grabber) + 62  [0x106a93e9e]
    +         9742 spdlog::details::async_log_helper::worker_loop()  (in karabiner_grabber) + 88  [0x106a93108]
    +           9742 spdlog::details::async_log_helper::process_next_msg()  (in karabiner_grabber) + 102  [0x106a93556]
    +             9742 spdlog::details::mpmc_bounded_queue<spdlog::details::async_log_helper::async_msg>::dequeue_for(spdlog::details::async_log_helper::async_msg&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)  (in karabiner_grabber) + 446  [0x106a93c8e]
    +               9742 std::__1::cv_status std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > > const&)  (in karabiner_grabber) + 117  [0x106a93d65]
    +                 9742 std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)  (in libc++.1.dylib) + 93  [0x7fff52aa3d43]
    +                   9742 _pthread_cond_wait  (in libsystem_pthread.dylib) + 732  [0x7fff54cf2662]
    +                     9597 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff54bb6e7e]
    +                     145 cerror_nocancel  (in libsystem_kernel.dylib) + 20  [0x7fff54bae780]
    9742 Thread_46446936
    + 9742 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff54cf0c5d]
    +   9742 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff54cf156d]
    +     9742 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff54cf16c1]
    +       9742 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>::*)(), krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>*> >(void*)  (in karabiner_grabber) + 62  [0x106a97a1e]
    +         9742 krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>::loop()  (in karabiner_grabber) + 107  [0x106a9763b]
    +           9742 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)  (in libc++.1.dylib) + 18  [0x7fff52aa3cb0]
    +             9742 _pthread_cond_wait  (in libsystem_pthread.dylib) + 732  [0x7fff54cf2662]
    +               9742 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff54bb6e7e]
    9742 Thread_46446938
    + 9742 start_wqthread  (in libsystem_pthread.dylib) + 13  [0x7fff54cf0c4d]
    +   9673 _pthread_wqthread  (in libsystem_pthread.dylib) + 1035  [0x7fff54cf106a]
    +   ! 9673 __workq_kernreturn  (in libsystem_kernel.dylib) + 10  [0x7fff54bb76da]
    +   69 _pthread_wqthread  (in libsystem_pthread.dylib) + 842  [0x7fff54cf0fa9]
    9742 Thread_46448362
      9742 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff54cf0c5d]
        9742 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff54cf156d]
          9742 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff54cf16c1]
            9742 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, krbn::receiver::receiver(krbn::device_grabber&)::'lambda'()> >(void*, void*)  (in karabiner_grabber) + 45  [0x106b1429d]
              9742 krbn::receiver::worker()  (in karabiner_grabber) + 322  [0x106b14452]
                9742 krbn::local_datagram_server::receive(boost::asio::mutable_buffer const&, boost::posix_time::time_duration, boost::system::error_code&)  (in karabiner_grabber) + 296  [0x106b15128]
                  9742 boost::asio::io_context::run_one()  (in karabiner_grabber) + 156  [0x106b158ac]
                    9742 boost::asio::detail::scheduler::run_one(boost::system::error_code&)  (in karabiner_grabber) + 281  [0x106b16a59]
                      9742 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)  (in karabiner_grabber) + 337  [0x106b16d51]
                        9742 boost::asio::detail::kqueue_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&)  (in karabiner_grabber) + 273  [0x106b17181]
                          9742 kevent  (in libsystem_kernel.dylib) + 10  [0x7fff54bb803a]

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        __psynch_cvwait  (in libsystem_kernel.dylib)        19339
        kevent  (in libsystem_kernel.dylib)        9742
        __workq_kernreturn  (in libsystem_kernel.dylib)        9673
        mach_msg_trap  (in libsystem_kernel.dylib)        7889
        IOConnectCallStructMethod  (in IOKit)        464
        free_tiny  (in libsystem_malloc.dylib)        146
        cerror_nocancel  (in libsystem_kernel.dylib)        145

@saagarjha
Copy link
Contributor

saagarjha commented May 25, 2018

Wow, that's a much more interesting sample!

operator new(unsigned long)

@tekezo, yeah, it looks like std::vector::emplace_back will have to be rethought as well. Really, everything that has significant weight from that trace needs to go.

@tekezo
Copy link
Member

tekezo commented May 25, 2018

Thank you for feedbacks!
I try to reproduce this issue by such stress testing.

Note:
If the issue happened by high CPU usage of multiple processes,
there are possibility of macOS limitation.
In such situation, generic processes are not assigned much processing time and
it causes a delay of key events arrival to karabiner_grabber.
I guess macOS is assigning extremely high priority to own event handling process (hidd) to avoid this problem.

@saagarjha
Copy link
Contributor

saagarjha commented May 25, 2018

I guess macOS is assigning extremely high priority to own event handling process (hidd) to avoid this problem.

You mean something like a higher nice value? I don't see anything out of the ordinary here for hidd, so is there some other mechanism that macOS might be using here?

If the issue happened by high CPU usage of multiple processes

I see this issue occur even when there aren't a whole lot of processes. I can try making a multithreaded test and seeing what happens…

EDIT: I created a multithreaded test case:

#include <pthread.h>

void *wait(void *foo) {
	while (1) {
	}
}

int main() {
	pthread_t threads[100];
	for (int i = 0; i < 100; ++i) {
		pthread_create(&threads[i], NULL, wait, NULL);
	}
	pthread_join(*threads, NULL);
	return 0;
}

This one causes a repeat to occur immediately (as in, the key event that comes from pressing enter to run the command gets held on to and repeated). Interestingly, the sample has very little of note:

Call graph:
    9749 Thread_27453668   DispatchQueue_1: com.apple.main-thread  (serial)
    + 9749 start  (in libdyld.dylib) + 1  [0x7fff5a107015]
    +   9749 main  (in karabiner_grabber) + 3531  [0x1045dd58b]
    +     9749 CFRunLoopRun  (in CoreFoundation) + 99  [0x7fff32342c33]
    +       9749 CFRunLoopRunSpecific  (in CoreFoundation) + 483  [0x7fff323041a3]
    +         9749 __CFRunLoopRun  (in CoreFoundation) + 1783  [0x7fff32304b27]
    +           9749 __CFRunLoopServiceMachPort  (in CoreFoundation) + 341  [0x7fff323057d5]
    +             9749 mach_msg  (in libsystem_kernel.dylib) + 60  [0x7fff5a24d724]
    +               9749 mach_msg_trap  (in libsystem_kernel.dylib) + 10  [0x7fff5a24e20a]
    9749 Thread_27453669
    + 9749 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebf9]
    +   9749 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff5a41f50d]
    +     9749 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff5a41f661]
    +       9749 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (spdlog::details::async_log_helper::*)(), spdlog::details::async_log_helper*> >(void*)  (in karabiner_grabber) + 62  [0x1045f7e9e]
    +         9749 spdlog::details::async_log_helper::worker_loop()  (in karabiner_grabber) + 88  [0x1045f7108]
    +           9749 spdlog::details::async_log_helper::process_next_msg()  (in karabiner_grabber) + 102  [0x1045f7556]
    +             9749 spdlog::details::mpmc_bounded_queue<spdlog::details::async_log_helper::async_msg>::dequeue_for(spdlog::details::async_log_helper::async_msg&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)  (in karabiner_grabber) + 446  [0x1045f7c8e]
    +               9749 std::__1::cv_status std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > > const&)  (in karabiner_grabber) + 117  [0x1045f7d65]
    +                 9749 std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)  (in libc++.1.dylib) + 93  [0x7fff5805cd43]
    +                   9749 _pthread_cond_wait  (in libsystem_pthread.dylib) + 732  [0x7fff5a420589]
    +                     9749 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff5a257a1e]
    9749 Thread_27453670
    + 9749 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebf9]
    +   9749 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff5a41f50d]
    +     9749 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff5a41f661]
    +       9749 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>::*)(), krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>*> >(void*)  (in karabiner_grabber) + 62  [0x1045fba1e]
    +         9749 krbn::async_sequential_dispatcher<krbn::async_sequential_file_writer::entry>::loop()  (in karabiner_grabber) + 107  [0x1045fb63b]
    +           9749 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)  (in libc++.1.dylib) + 18  [0x7fff5805ccb0]
    +             9749 _pthread_cond_wait  (in libsystem_pthread.dylib) + 732  [0x7fff5a420589]
    +               9749 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7fff5a257a1e]
    9749 Thread_27453701
    + 9749 thread_start  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebf9]
    +   9749 _pthread_start  (in libsystem_pthread.dylib) + 377  [0x7fff5a41f50d]
    +     9749 _pthread_body  (in libsystem_pthread.dylib) + 340  [0x7fff5a41f661]
    +       9749 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, krbn::receiver::receiver(krbn::device_grabber&)::'lambda'()> >(void*, void*)  (in karabiner_grabber) + 45  [0x10467829d]
    +         9749 krbn::receiver::worker()  (in karabiner_grabber) + 322  [0x104678452]
    +           9749 krbn::local_datagram_server::receive(boost::asio::mutable_buffer const&, boost::posix_time::time_duration, boost::system::error_code&)  (in karabiner_grabber) + 296  [0x104679128]
    +             9749 boost::asio::io_context::run_one()  (in karabiner_grabber) + 156  [0x1046798ac]
    +               9749 boost::asio::detail::scheduler::run_one(boost::system::error_code&)  (in karabiner_grabber) + 281  [0x10467aa59]
    +                 9749 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)  (in karabiner_grabber) + 337  [0x10467ad51]
    +                   9749 boost::asio::detail::kqueue_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&)  (in karabiner_grabber) + 273  [0x10467b181]
    +                     9749 kevent  (in libsystem_kernel.dylib) + 10  [0x7fff5a258bf2]
    9749 Thread_27455000
      9749 start_wqthread  (in libsystem_pthread.dylib) + 13  [0x7fff5a41ebe9]
        9749 _pthread_wqthread  (in libsystem_pthread.dylib) + 1035  [0x7fff5a41f009]
          9749 __workq_kernreturn  (in libsystem_kernel.dylib) + 10  [0x7fff5a258292]

@tekezo
Copy link
Member

tekezo commented May 27, 2018

I changed the karabiner_grabber process priority at v12.0.12.
Please confirm the latest beta version.
(You might have to restart your Mac once to apply the change.)

The main change is adding ProcessType: Interactive into launchd.plist.
6d1e38e
This property assigns higher priority to karabiner_grabber.

Detail

I confirmed your issue with this code.
https://github.com/tekezo/Karabiner-Elements/blob/master/appendix/stress_testing/main.cpp
https://github.com/tekezo/Karabiner-Elements/blob/master/appendix/stress_testing/run.bash

I also confirmed the problem is reproduced without Karabiner-Elements at high concurrency. (e.g., 100 processes on my machine)

Thus, this issue with high CPU usage is a macOS architecture limitation.
However, we can increase the threshold as possible.

The above change is one of the way to do it.

@ghost
Copy link

ghost commented May 27, 2018

WOOT! This seems to have fixed the issue, at least for me (And I was able to reproduce it VERY easily). I pegged my CPU to the point where typing is being delayed and I'm not getting any repeated characters, yay!!!

@saagarjha
Copy link
Contributor

Yes, this seems to have fixed it! The fact that you needed to add set the daemon to Interactive might explain the odd sample I had above: launchd was just pausing karabiner_grabber, which is why it wasn't doing any meaningful work.

Anyways, I'll use this for a couple days and use it as I was previously; I'll let you know how it continues to perform. Great work!

@ryantuck
Copy link

This does seem to be entirely fixed (running v12.1). Thank you so much @tekezo 🙌

@ignaciovazquez
Copy link

ignaciovazquez commented May 31, 2018 via email

@saagarjha
Copy link
Contributor

saagarjha commented Jun 3, 2018

Ok, so after using this for a week the issue has mostly disappeared, except for one infrequent case involving the function keys (the brightness and volume keys are the easiest to see, because their results are immediately visible on the screen). Interestingly, in this case, the load is rather low until the key is pressed, when it spikes and causes the key to repeat. It's pretty rare and hard to reproduce, which has the upside that it's not really annoying, but the downside that it's hard to fix; let me know if you can figure out what's going on here.

@lumie1337
Copy link

lumie1337 commented Jul 7, 2018

I just ran into this issue again (under high multi-threaded cpu load) even though the grabber process has a high niceness level now. What was interesting was the fact that while the cpu load stayed relatively stable, the issue only appeared very regularly about every 5 seconds (either repeating or not reacting to input for about half a second). Without karabiner enabled input worked flawlessly.

Interesting however: if i put load on the cpu by spawning a bunch of yes > /dev/null processes, it does not trigger this issue at all. So I would say it is probably related to IO / Kernel activity of some sort that still triggers the issue.

Did some checking and it appears there are some apis to get real time behaviour for threads (Interactive only increases process priority with no guarantees)
Check out how chrome handles it for low latency audio (in SetPriorityRealtimeAudio, especially THREAD_TIME_CONSTRAINT_POLICY): https://github.com/chromium/chromium/blob/367a8a06efde403659f13a8f2ca6f16b6b5ecf92/base/threading/platform_thread_mac.mm

@stale
Copy link

stale bot commented Sep 5, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Sep 5, 2018
@stale stale bot closed this as completed Sep 12, 2018
wincent added a commit to wincent/wincent that referenced this issue Nov 24, 2018
Trying to work around the issue noted in the comment:

  pqrs-org/Karabiner-Elements#1645

Possibly also related to:

  pqrs-org/Karabiner-Elements#545

Seems that under high CPU load (which might be happening at wake-up
time) Karabiner may be exacerbating a tendency of the OS to jam the
keyboard. I've had a number of hard-resets as a result of this lately:

  https://wincent.com/snippets/1159

The commands used to enable/disable here taken from:

  https://github.com/tekezo/Karabiner-Elements/blob/b38990db3/src/share/launchctl_utility.hpp#L18-L43
@ivnsch
Copy link

ivnsch commented Feb 29, 2020

I want to add that I've also a repeated key problem, but it's not because of Karabiner (I have it installed but it wasn't always running when it happened).

Also, it doesn't seem to be related with a specific keyboard (or at least the "external" key triggers), as I've tried both disabling the built in keyboard with karabiner and disconnecting my external keyboard and it doesn't fix it.

@AndiDog
Copy link

AndiDog commented Feb 29, 2020

For me, the issue still shows and only applies to filtered devices. I have key remaps only for the internal MacBook keyboard, and only that one is affected. It's mostly happening when opening the laptop and filling the password field.

@svicalifornia
Copy link

This issue is happening for me, but only with my external Apple Bluetooth keyboard. When it happens, I'm forced to switch back to my M1 MacBook's built-in keyboard, and then I restart the MacBook to get the external keyboard to work properly without key repeats.

@Ken0sis
Copy link

Ken0sis commented Jun 27, 2021

I'm having the same problem, and with external bluetooth keyboard, similar to svicalifornia.

@clicseo
Copy link

clicseo commented Dec 22, 2021

Im also having this issue. With built-in keyboard. As soon as I quite Karabiner no more problems. It only happens in Apple apps? I have the keys repeated on Terminal or Safari quite often (once a minute almost) but I can't get it repeated while writing on Sublime.

@sichinumi
Copy link

sichinumi commented Mar 15, 2022

Confirming that I still see this issue multiple times a day in 2022. In addition, the repeat is sometimes actually delayed until after other keys are pressed. For example if I type the word "disappoint" it might show up as "disappointo" as the "o" is duplicated but not until after the "int" characters are registered. Usually the repeat is clustered, like "disappooint".

I've seen this happen with clusters of keypresses, sometimes up to 3. Perhaps totally coincidentally, it seems to happen the most with the "O" key involved. The number of repeats is usually 1 but it can be more than that, I've seen keys repeat 3 times before.

Symptoms seem to be worse when CPU-intensive apps are open like video streaming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests