bug: hererocks build failed because "python: command not found"

[kaix-canva]

Did you check docs and existing issues?

• I have read all the lazy.nvim docs
• I have searched the existing issues of lazy.nvim
• I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)

NVIM v0.10.0

Operating system/version

22.04.1-Ubuntu

Describe the bug

After fresh installation, I can see the following error:

    ● hererocks 0.07ms ✔ build
        /bin/bash: line 1: python: command not found

It looks like the offending line happens here:

lazy.nvim/lua/lazy/pkg/init.lua

Line 124 in 9005e8e

"python",

I think we'll need to check the availability of python before we start, and many people will have python3 in their path instead of python.

Steps To Reproduce

Install neovim with latest version of lazy. Open an empty file, lazy screen will pop up showing the aforementioned error.

Expected Behavior

Nothing should appear.

Repro

No response

[kaix-canva]

Also, after I've symlinked python3 to python locally, the build failed again. This time because missing readline/readline.h:

luaconf.h:275:10: fatal error: readline/readline.h: No such file or directory

I had to install it on my machine to finally get the plugin built:

sudo apt-get install libreadline-dev

[max397574]

oh true lua headers can be another issue
🤔 at this point it might be worth writing something just to install luarocks on every system either with different tools and package managers or just binaries with sth similar to mason

[ThaiDuy12345]

Having the same issue when hitting Update for all plugins

[folke]

Sigh, luarocks really is pretty fragile...
On Windows it's even worse.
Will make some changes...

[folke]

What if I'd just disable automatic luarocks packages completely?
And then if a certain plugin needs luarocks, you need to manually add {"foo/bar.nvim", build = "rockspec"}

[max397574]

that just makes it a pain for plugin authors and users ig
here people can work together to make a nice solution which works universally

otherwise this will make plugin authors think twice if they really want to use luarocks (or just be a pain for the ones who need things from there) and also requires users of such plugins to install things just themselves

perhaps @mrcjkb and @vhyrro could provide some help here with their experiences from luarocks.nvim and rocks.nvim things

[kaix-canva]

What if I'd just disable automatic luarocks packages completely? And then if a certain plugin needs luarocks, you need to manually add {"foo/bar.nvim", build = "rockspec"}

I think that's a sensible approach. I understand the intention, but I personally feel as a neovim plugin manager, we should not try to install a lua package manager especially when no binary distribution is provided for the various platforms we strive to support.

[folke]

that just makes it a pain for plugin authors and users ig
A plugin can just add a lazy.lua file with the build = "rockspec".

The biggest issue right now is plugins that also have a rockspec where they define dependencies to other Neovim plugins.

There's currently no way I can resolve these deps to the correct github paths. I have no idea how to do that (without using luarocks). Parsing a rockspec is easy though.

Like this one https://github.com/mfussenegger/nvim-dap-python/blob/master/nvim-dap-python-scm-1.rockspec

[max397574]

that just makes it a pain for plugin authors and users ig

A plugin can just add a lazy.lua file with the build = "rockspec".

I'm talking about the fact that plugin authors are responsible for the fact that their plugin needs an additional system dependency
and they'll get issues because of that and users won't use the plugin because of that or will have the pain of installing luarocks manually themselves to use the plugin

The biggest issue right now is plugins that also have a rockspec where they define dependencies to other Neovim plugins.

that shouldn't be an issue though if these are on luarocks right?
I don't see why you need to get a github path then

[Bart97]

Is there a workaround to disable building hererocks for now? I don't need this functionality, but I really need to get Lazy working again.

[Ax51]

Is there a workaround to disable building hererocks for now? I don't need this functionality, but I really need to get Lazy working again.

Hi! Also faced with this issue. These instructions worked for me:

So try to set lazy.opts.rocks.enabled to false and issue should be fixed:

[max397574]

ig with the rocks or pkg config you should be able to
https://lazy.folke.io/configuration

[mrcjkb]

In case this helps:

In rocks.nvim, we have a bootstrap script that works quite well on unix and doesn't depend on hererocks.
We also have luarocks itself as a dependency (you can install it with luarocks), so we can pin the version.

Our bootstrap script doesn't work with non cygwin Windows yet. It'll probably use the install.bat when it does eventually.

It's probably a good idea to silently disable luarocks packages if no luarocks executable is found and it can't be bootstrapped.

[Bart97]

Is there a workaround to disable building hererocks for now? I don't need this functionality, but I really need to get Lazy working again.

Hi! Also faced with this issue. These instructions worked for me:

So try to set lazy.opts.rocks.enabled to false and issue should be fixed:

Thanks, this works for me

[folke]

With hererocks disabled:

hererocks enabled:

Similar, but when lazy tries installing a plugin:

These errors should probably be more than enough for users?

[max397574]

I think this is a big improvement
the best solution imo would be to try bootstrapping it and disabling it if not found like mrcjkb suggested here #1548 (comment)

Because ig that will work for a lot of users (even more if we'd extend the script to use hererocks as a fallback) which would just lead to less failed installs

[folke]

Can't even get luarocks to work with my mingw installation in Widnows.
Had to install a bunch of additional things with scoop, but still fails with a non-descriptive error:
Warning: Failed searching manifest: Failed downloading https://nvim-neorocks.github.io/rocks-binaries/dev/manifest-5.1 - failed downloading https://nvim-neorocks.github.io/rocks-binaries/dev/manifest-5.1
Cloning into 'nvim-dap'...
sed: Bad locale: LC_CTYPE=C.UTF-8
sed: Bad locale: LC_CTYPE=C.UTF-8
sed: Bad locale: LC_CTYPE=C.UTF-8
C:/Program Files/Git/mingw64/libexec/git-core\git-submodule: line 613: cmd_: command not found

    Error: Failed installing dependency: https://luarocks.org/dev/nvim-dap-scm-1.rockspec - Failed to fetch submodules.

[folke]

Does anyone know how I can find the rockspec for a rock name?

[folke]

When I restart Neovim, luarocks also sometimes won't launch at all.
Then restarting a couple of times and it works again.
With "Won't launch" I mean vim.uv.spawn returns nil
So weird.

[folke]

The whole luarocks ecosystem is a mess.
tbh, I've never used a package manager that works as terrible as this.
The big issue is that since I've added support to lazy, I will now need to support that mess.

I'll leave it as is for now, but I'm strongly considering to make luarocks opt-in.

[folke]

Plugins should really just use a lazy.lua file to specify their dependencies and their configuration.
Nothing else is needed.
With luarocks it's not even possible for a plugin to declare that it need setup or that it can be lazy-loaded.
luarocks as a dependency in a lazy.lua file could still make some sense and will work, but not for plugins

[max397574]

I think the most important thing to be gained from luarocks is installing things which aren't neovim plugins like toml or e.g. fzy from luarocks
would that still work from a lazy.lua file?

[mrcjkb]

Does anyone know how I can find the rockspec for a rock name?

Example for neorg 8.7.1-1

https://luarocks.org/manifests/neorg/neorg-8.7.1-1.rockspec

You can find versions in the manifest:

https://luarocks.org/manifest-5.1

[folke]

@mrcjkb that requires the user name of the package to be known as well, but that's something that I don't know.

That nvim-dap-python package for example depends on nvim-dap.

I would like to be able to ideally resolve nvim-dap to its GitHub repo.

If a Neovim plugin only has a simple build step and depends on only neovim plugins, then I can skip luarocks altogether in that case.

[folke]

I think the most important thing to be gained from luarocks is installing things which aren't neovim plugins like toml or e.g. fzy from luarocks
would that still work from a lazy.lua file?

That use case makes sense and this should of course still work.

I just want to prevent needing to use luarocks for things like telescope, plenary, nvim-dap and others that don't really need luarocks to work.

Ideally luarocks should only be used when needed.

[mrcjkb]

@folke sorry I posted the wrong URL

https://luarocks.org/nvim-dap-scm-1.rockspec works too.

The problem is that not all rockspecs have a git source. Some (especially semver releases) might have a zip archive as the source.

I just want to prevent needing to use luarocks for > things like telescope, plenary, nvim-dap and others > that don't really need luarocks to work.

Ideally luarocks should only be used when needed.

I think if luarocks packages aren't first class citizens then luarocks support should definitely be opt-in (and plugins that require it could enable it in a lazy.lua) file.

[folke]

Sweet! Thanks!

[noahlias]

I found an error
when I rebuild neorg plugin with command lazy build neorg

checkhealth:

But I have installed neorg in lazy-rocks with previous version lazy.nvim

And the neorg plugin works, How to fix this wierd thing ?

[folke]

Take a look at https://github.com/rockerBOO/awesome-neovim and find me three plugins without nvim in the name :)

It also doesn't really matter, since lazy is capable of installing any luarock url.
So I could effectively parse the full manifest and take over all dependency resolving on the lazy side, and just resort to luarocks for complex builds, but that's not really needed and would be too much.

The issue I'm solving is where some Neovim plugins are listed as a dependency in a luarock, but might also be in your lazy spec somewhere. Currently that would be two different things on your package path. This can potentially lead to issues.

Pretty much all of the Neovim plugin rockspecs I looked at, had either no dependencies or only Neovim plugin dependencies.

So for those cases, the rockspec should only be used to determine dependencies and let lazy handle the rest.

For more advanced plugins like Neorg, luarocks would still be used for some of its dependencies, like the tree-sitter-norg stuff.

[idk-r-n]

I didn't realize how messy this was. It is obvious when looking at a plugin that it uses a Lua rock?

[max397574]

yes it is because of the rockspec

[max397574]

Take a look at https://github.com/rockerBOO/awesome-neovim and find me three plugins without nvim in the name ツ

aren't in the list bug they are there:

[folke]

So cmp basically. I'll add those as well

[max397574]

well this will also be the case for the completion plugin I'm currently working on (and perhaps there are also other things as well)

[folke]

Just one plugin aff7ee8

[folke]

But again, that doesn't really matter that much. It will work nonetheless

[folke]

The new code is live.

For neorg, you'll now see all it's depenencies and their depedencies as separate plugins in :Lazy.
Only the tree-sitter-norg one will be built by luarocks. The others are all managed by lazy.

[folke]

My issue with nvim-dap-python and some others is now also resolved...

[max397574]

just one thing (would take too much time to go figure that out in the code)
what happens if two plugins have the same dependency with different versions?

not that I had an example where it mattered
just interested in knowing

[max397574]

also just for clarification because of the recent changes and discussions about opt-in or opt-out

lazy does by default now parse pack-spec, rockspec and lazy.lua
rockspec dependencies are installed like normal plugins if they don't have special build steps. Only if they have special build steps they need the luarocks binary correctly installed
If there are such plugins luarocks will be tried to be used by default ootb without any additional user config (but could be disabled)

is that correct?

[folke]

just one thing (would take too much time to go figure that out in the code)
what happens if two plugins have the same dependency with different versions?

Good point, versions are currently silently ignore. ha.
Still need to add that part, but will add an option to disable rockspec versions.

rockspec dependencies are installed like normal plugins if they don't have special build steps. Only if they have special build steps they need the luarocks binary correctly installed

If there are such plugins luarocks will be tried to be used by default ootb without any additional user config (but could be disabled)

Yes, correct.

With the latest changes, if a user didn't need luarocks before v11, they won't need it now. That was the main thing I wanted to fix.

[mrcjkb]

The whole luarocks ecosystem is a mess.
tbh, I've never used a package manager that works as terrible as this.

I didn't realize how messy this was. It is obvious when looking at a plugin that it uses a Lua rock?

To be fair, the luarocks ecosystem itself isn't a mess. It just isn't intended to be used in a way that doesn't treat lua rocks as first class citizens.

To elaborate:

luarocks installs rocks into a tree that separates library (lua, lib) and runtime (share/<name>/<version>/{plugin, ftplugin, ..., whatever}).
This allows to make Lua libraries available without impacting startup time in any meaningful way, and to add runtime data as needed.

Some Nvim libraries double as plugins and libraries, but don't need to be plugins for the average user.
For example, plenary.nvim is mostly a library, but also a plugin that creates commands for testing Neovim plugins (useless if you're not a plugin developer).

The idea of using luarocks to manage plugins as first class citizens is:

• Users should not have libraries their plugins depend on pollute Nvim with unexpected behaviour or unnecessarily impact startup time.
• The behaviour should only be added if users explicitly declare that they want that to happen (by installing it as a plugin).
• Extensions of plugins should be able to declare version constraints on the plugin they're extending.

In our [nvim-neorocks org] experience, this works quite well.
luarocks even has the ability to load different versions of the same dependency.

The main PITAs we still have are;

• macOS and Windows package managers being sh*te
  (which is why we're working on better ways to bootstrap luarocks and ensure a compatible version is installed - and have also been working on a rewrite¹).
• Slow bulk installations because luarocks doesn't yet have good mechanisms to allow parallel builds without potentially corrupting shared dependencies.

... which is why I was a bit surprised that this was merged and released so quickly.

With luarocks it's not even possible for a plugin to declare that it need setup or that it can be lazy-loaded.

There exists no standard for declaring such things, so it wouldn't be very useful for diversity.
There is, however, a very simple solution to this problem that works independently of the package manager.

Footnotes

1. It's too soon to say if we will use it to replace luarocks. ↩

[mrcjkb]

I just deployed a new Github action that fetches the luarocks root manifest and then for each rock that contains nvim in the name (currently 177 rocks), it fetches the rockspec.

@folke many neovim plugins have 'neovim' or 'nvim' in the description.labels list.
Perhaps checking that could make the GitHub action more reliable.

[folke]

The biggest issue is tooling as far as I can see.
Things like npm, rust etc all just work on Linux, Macos, Windows.
For me on Arch, luarocks also works great, but in any other system I tried, it was a big pita.
I still couldn't get it to work on Windows for example...
And installing lua5.1 on Macos seems impossible.

Slow bulk installations because luarocks doesn't yet have good mechanisms to allow parallel builds without potentially corrupting shared dependencies.

Lazy does, and also now does it for luarocks, since the those will be in separate trees.

... which is why I was a bit surprised that this was merged and released so quickly.

As I said before, with lazy v11.0, all of the sudden, some of user's plugins needed luarocks.
If a uses didnt need luarocks before v11, then they shouldn't need it after.
This is now fixed with my recent changes.

[folke]

many neovim plugins have 'neovim' or 'nvim' in the description.labels list.
Perhaps checking that could make the GitHub action more reliable.

This would require to download all rockspecs, unless there's a repo somewhere that contains them all?

[mrcjkb]

Ah, I had assumed you were downloading the rockspecs already.
Yep, that might hit rate limits pretty quickly. Hadn't thought of that.

I guess you could have a cron job that crawls a subset of https://luarocks.org/manifest-5.1, but that might be overkill for the few plugins that don't have nvim in the name. It would probably be easier to just add the popular ones manually.

[folke]

I'm currently using https://luarocks.org/manifest.
Should I use the 5.1 version instead?

[folke]

Closing, since op's original issue was also fixed.

[mrcjkb]

I'm currently using https://luarocks.org/manifest. Should I use the 5.1 version instead?

Probably not a bad idea. 5.2..4 rocks won't work with neovim anyway.

[mehalter]

cmp

Also the neotest ecosystem:

https://github.com/nvim-neotest/neotest - is published to luarocks
https://github.com/mrcjkb/neotest-haskell/blob/master/neotest-haskell-scm-1.rockspec - is not correctly converted to lazy dependencies

[folke]

@mehalter added!
6d60dc3