.fnl files not automatically compiling on save

[Rotbjorn]

Hello 👋! I'm having issues with files not being automatically compiled on save inside of my neovim directory.

# Neovim appimage through WSL
NVIM v0.9.1
Build type: Release
LuaJIT 2.1.0-beta3

Relevant files:

;; ~/.config/nvim/.nfnl.fnl
{:source-file-patterns ["fnl/**/*.fnl"]}

;; ./fnl/test.fnl
(vim.notify :hello)
{ :hello 123 }

-- ./lua/plugins/nfnl.lua
return {
     "Olical/nfnl",
     ft = "fennel",
}

Saving does not generate the file lua/test.lua, however require [[nfnl]]['compile-all-files ]() does work as expected. Lazy shows nfnl as loaded when opening a fennel file.

:au * fennel
# shows `nfnl-setup  FileType`

~Also unrelated: is there a way to specify a target directory for the generated files?

[Olical]

If you use {:source-file-patterns ["fnl/**/*.fnl" "fnl/*.fnl"]} does that help? My thinking is there could be a difference in how the glob works when you use compile-all-files compared to the autocmd that gets created. Both use the same glob string but might interpret it differently.

[Olical]

And no option to pick a output dir yet, everything is based on a string substitution from fnl -> lua.

Where would you want to send the file to instead? I need to decide if some config data or a function you can define yourself will be the best option. Shouldn't be much work really, the per-project config system makes this sort of thing kinda easy to add, just have to work out the right way to express it.

[Olical]

Oh and one more question! Does writing to files under fnl/foo/bar.fnl compile? Is it just fnl/foo.fnl that won't compile? And what happens if you just use the default config with {}?

[Rotbjorn]

If you use {:source-file-patterns ["fnl/**/*.fnl" "fnl/*.fnl"]} does that help? My thinking is there could be a difference in how the glob works when you use compile-all-files compared to the autocmd that gets created. Both use the same glob string but might interpret it differently.

That fixed it! Files directly under ./fnl now get correctly compiled.

Oh and one more question! Does writing to files under fnl/foo/bar.fnl compile? Is it just fnl/foo.fnl that won't compile? And what happens if you just use the default config with {}?

Tested creating both fnl/foo/bar.fnl and fnl/foo/baz/foobar.fnl, both got compiled correctly with {:source-file-patterns ["fnl/**/*.fnl"]}

The default config ({}) correctly compiled fnl/test.fnl, fnl/foo/bar.fnl, fnl/foo/baz/foobar.fnl, and root.fnl in the root directory by the .nfnl.fnl file.

[Rotbjorn]

And no option to pick a output dir yet, everything is based on a string substitution from fnl -> lua.

Where would you want to send the file to instead? I need to decide if some config data or a function you can define yourself will be the best option. Shouldn't be much work really, the per-project config system makes this sort of thing kinda easy to add, just have to work out the right way to express it.

I don't know the philosophies behind nfnl, but specifically for my neovim config, I would prefer a system where I can:

1. direct the compiled output to some other folder, perhaps some folder under stdpath('data') and,
2. add that path to package.path (or runtimepath?) to let lua load the modules.

That way my neovim config folder just contains fennel code + bootstrapping lua code that adds Lazy + nfnl etc. that then consequently compiles all fennel code to stdpath('data') . '/nfnl' on clean installs of Neovim, + giving the appearance of Neovim natively running fennel code since you don't see any lua artifacts.

[Olical]

Okay cool, so this issue now is to make sure the glob patterns work as expected when compiling things manually and within the autocmd. Or maybe it just won't work and I'll need to make sure users always include the middle layer of the glob like "fnl/**/*.fnl" "fnl/*.fnl" since it's kinda ambiguous and works differently in different systems. Maybe the best bet is just to suggest you are more explicit with your globs so the lowest common denominator system still works as expected with it. And it's a noop for the systems that DO read fnl/**/*.fnl as including fnl/*.

I also created #6 to track the extra config.

[Olical]

And the philosophy behind nfnl needs expanding somewhere to convince people about the design. This is from years of fiddling with nvim+fnl+plugins+dotfiles, so I think I'm at a local maximum but I know it's still a sea of opinions 😄

From the readme:

In the case where you're absolutely adamant that you need to .gitignore your compiled Lua output, this can be used after you git pull to ensure everything is compiled. I strongly advise committing your Lua for performance and stability reasons however.

This project was designed around the principal of compiling early and then never needing to compile again unless you make changes. I thought long and hard about the tradeoffs so you don't have to.

I'll expand on this more at a later date but essentially I want lua+fennel committed alongside each other for startup speed, predictability and the ability to rollback / forward to any commit of your dotfiles and have everything Just Work.

[Rotbjorn]

I think pushing build artifacts is just unorthodox in a lot of people's minds including mine, but I think it's a fair design choice.

I believe lazy-lock.json¹ solves the reproducibility problem when switching between commits, but I dont think that would interface that great with the system I described earlier, in contrast to just committing lua/* and let Neovim do its thing.

Your solution does Just Work™², and I'm only against it because I don't like pushing artifacts in order to keep the repo clean - which is ultimately trivial and doesn't really net you any benefits apart from a clean repo.

Footnotes

1. if you use lazy, that is ↩

2. and I've seen your vimconf talks, so I'm confident you know better than me ↩

[Olical]

I think pushing build artifacts is just unorthodox in a lot of people's minds including mine, but I think it's a fair design choice.

Normally, yes, but when your git repo is your source and your package registry / distribution it becomes tricky to separate them. If this were Java, of course you wouldn't commit a .class, but you have maven repositories for that, clojars for Clojure, npm for JavaScript, pypi for Python, LuaRocks for Lua etc etc.

In a world where Source == Package (vim plugins) we have to consider the repo itself the artifact. The alternative is to bundle in enough logic and code to automatically compile everything JIT when it's first run.

This means more code being loaded and run on startup with I know from experience the Neovim community tends to be deathly allergic to. So that's why I fall on this side of the fence, treat the repo as the artifact, avoid any code or compilers from running at startup, have the exact system you expect if you check out ANY commit (since the repo IS the artifact).

Yes in other contexts it'd be icky, but to have a robust, fast and good system we just need to move past that thing that's yucky under different circumstances. It's a matter of pragmatism and I think I swallowed the pill so long ago I forget that most others will need convincing 😅

And like I've documented in the readme, you can tell GitHub it's compiled code and then it doesn't even include it in their analysis / summary of your repo, it still counts as a Fennel repo. Just look at the one for this repo, there's SO much more Lua in here than any other language, but it doesn't really feel like it at a glance which is nice.

I'm only writing all of this as context to link to later when I get this question / point another 100 times 😬 so even if you're convinced, this may be helpful for others in the future.

Now I'm going to step out for a sec then implement the "write the Lua to another directory" feature, but it'll come with a caveat of the user needs to know what they're doing. They need to then ensure the compiler is run at the right times and the Lua is written somewhere that can be loaded by Neovim easily. So it's kind of a power user thing for people who want a very specific experience.

I still advice against it and think it's far more trouble than it's worth, but I'll do my best to support it.

[Olical]

I've made this configurable by allowing you to override, wrap or replace the function that decides how a fnl path should be translated to a Lua one. You can do whatever you want with this. I have included an example which is almost what you need, just needs a ~/... path instead of a sub directory I think? Although you may have to expand ~ because that doesn't work with many nvim fs features.