Slow completion on macOS Big Sur

[laserscout]

Hello,

After upgrading to macOS 11 I had issues with fish shell and auto completion. It was fixed on fish here but it seems that I'm having a similar issue with company-shell; as completion is always slow on .sh files.

Could you help me resolve this issue?

Thanks

[Alexander-Miller]

It's unlikely your problem has anything to do with company-shell - it talks to the shell just once, when you try to complete for the first time. After that a cached list of completions is used.

You should use the builtin profiler to investigate what Emacs is really doing.

[laserscout]

I have pinpointed the issue.

It's a known issue that since macOS implemented read-only filesystem for system directories, the whatis database is missing and calls to whatis or apropos take a long time; eg MR for circumventing this in fish shell completion. That affects company-shell during the call of company-shell--meta-string.

One way of fixing that would be to create custom whatis and apropos scripts on the host OS that create a whatisdb in a rw path. I have implemented that successfully, but I'm not sure if it's appropriate to "officially" have that dependency for company-shell.

What do you think @Alexander-Miller ?

[memeplex]

I'm also experiencing this as reported in #16, some debugging revealed that company-shell--meta-string is slowing down the completion and this because of a call to whatis.

One thing I'd like to remark is that this isn't limited to fish shell completions, I use bash and don't have the fish backend enabled at all.

Given that this makes the extension almost unusable (I'm not exaggerating, it's too easy to freeze emacs by pressing TAB in the wrong place), even fallbacking to another backend (presumably capf) in case there already is a prefix in the line to be completed would be preferable in macOS.

[dsedivec]

Same here. It seems to be a known problem that whatis and makewhatis are broken on macOS, and fixing it seems less than totally easy (possible solutions include "exclude /usr/bin and such from makewhatis path" or "reboot with SIP disabled"). Nearly every completion would hang up Emacs for multiple seconds, just as running whatis foo from my terminal took several seconds to complete. I just neutralized the function in question from company-shell:

(defun my:company-shell-dont-run-whatis-on-macos ()
  nil)

(with-eval-after-load 'company-shell
  (when (eq system-type 'darwin)
    (advice-add 'company-shell--meta-string :override
                #'my:company-shell-dont-run-whatis-on-macos)))

[Alexander-Miller]

Sorry for the late response guys, the email got buried under a bunch of new stuff.

One way of fixing that would be to create custom whatis and apropos scripts on the host OS that create a whatisdb in a rw path. I have implemented that successfully, but I'm not sure if it's appropriate to "officially" have that dependency for company-shell.

I wouldn't want to include this in company-shell because I don't use macs, so it'd be something I don't really understand and I would always have to rely on someone who does whenever something needs to change. I'd be okay with adding custom variables for cases like this. A link to such a script would be great too, so I could link for from the readme and eldoc.

I just neutralized the function in question from company-shell:

That's also something a defcustom can take care of.

[laserscout]

A self contained solution would be generating, updating and using a whatis-db from a non-read-only filesystem. Such a solution might be beneficiary for all OSs as it might be faster and ensure the whatis-db is up to date.

Question is... Do we want to incorporate such a functionality in this package?

Alternatively, macOS specifically can depend on a system wide solution to this problem, but how can that be communicated to all users?

A system-wide solution can look something like:

Make the system commands look in one location

cp /usr/bin/apropos /usr/bin/whatis /usr/local/bin
sed 's:for d in /var/cache/man $manpath /usr/lib:for d in /var/cache/man #$manpath /usr/lib:' /usr/local/bin/apropos
sed 's:for d in /var/cache/man $manpath /usr/lib:for d in /var/cache/man #$manpath /usr/lib:' /usr/local/bin/whatis
sudo mkdir -p /var/cache/man
sudo chmod 777 /var/cache/man
man --path | xargs /usr/libexec/makewhatis -o /var/cache/man/whatis

Use man-db from brew

brew install man-db
echo 'PATH="/usr/local/opt/man-db/libexec/bin:$PATH"' #do persistently 
sudo mkdir -p /var/cache/man
sudo chmod 777 /var/cache/man
mandb

[Alexander-Miller]

Question is... Do we want to incorporate such a functionality in this package?

No. This package is a company-mode backend. Creating shell scripts to copy around items in /usr/bin is well beyond its intended scope. The fix is ultimately a workaround for macos-specific bugs, an operating system that I have no familiarity with, nor affinity for. This is not a maintenance burned I want to take upon myself.

For now my plan would be the following:

• add a defcustom to skip the meta annotations
• add a cache layer for meta annotations to avoid contstant lookups with whatis
• experiment with precomputing every meta string in the initial cache build, pick the less annoying option

[laserscout]

add a cache layer for meta annotations to avoid contstant lookups with whatis

That's a much better formulation of what I tried to say as the first option.

IMO it's the best course of action.

I would be more than happy to help you with this; at least with testing on macOS.

[dsedivec]

In case it helps for planning, whatis ls on one of my Macs takes 2 s and 6 s on another. The difference might be attributed to having more software installed on the latter but I'm not positive. That was best of five or six runs in a row.

[Alexander-Miller]

Done. You can now use company-shell-dont-fetch-meta to avoid the whatis lookup.

The meta strings are cached as well now, but that cache will not be pre-computed - it would take minutes for hundreds of items, even without the slowdown.