Discussion: Driving this project forward?

[jsmestad]

Background

This is the best place I could think of to post this sort of thing.

I was speaking with @axelson about this project on Elixir Slack today and we both agreed this project seems to be suffering due to no clear person driving it forward. If we look at the discussions about changing CI tooling in #458 there are some great ideas but no one is pushing it over the finish line.

Questions

• Can we update the README to detail out who the maintainers of the project are?
• Should we publish a ROADMAP.md to show areas we think need to be added or revised?
  • I have seen in several PRs there has been talk about things needing to be re-done, but no official list of things we can point to.
• Would shifting this project to being a "one stop shop" for Elixir in Emacs better drive adoption than the current separate extensions? Looking at how evil-collection works where it's distributed as one package, but you can still pick-and-choose functionality.
• In terms of licensing, do we want to enforce any sort of Emacs-compatible license in hopes of being merged into Emacs core OR do we want to keep this separate long term so it's not license-restricted in what it can do?
• How should we get more volunteers to step in and help maintain the project?
• How should we document how to get an entire Elixir mode configured?
  • Should we publish instructions like in this ElixirForum thread?
• Should we combine our efforts with what ElixirLS is doing? Should we shift to a "we expect ElixirLS"?
• Do we want to support polymode for something like inline HTML templates

Closing thoughts

Back earlier this year I spoke with @axelson and @Trevoke about how to help contribute to this project. Not without quite a bit of effort on my part, I was unable to figure out who to speak with or how to go about helping push this project forward. I'd love to close the gap between this extension and what VSCode is doing.

[axelson]

First off, thanks for getting this discussion started! There's some good questions to ponder in here. Here's some initial thoughts:

Can we update the README to detail out who the maintainers of the project are?

I think this makes sense. I believe it would be @victorolinasc, @Trevoke, and myself. Not sure if I'm missing someone because I don't see a way for me to list out all the members

Should we publish a ROADMAP.md to show areas we think need to be added or revised?

I think a simple issue might be easier to manage, and we could pin it in the issues list

Would shifting this project to being a "one stop shop" for Elixir in Emacs better drive adoption than the current separate extensions? Looking at how evil-collection works where it's distributed as one package, but you can still pick-and-choose functionality.

I'm not sure about how this would end up working. My concern is that this would stretch our (as in maintainer's + community's) resources too thin. But perhaps I'm wrong about that.

In terms of licensing, do we want to enforce any sort of Emacs-compatible license in hopes of being merged into Emacs core OR do we want to keep this separate long term so it's not license-restricted in what it can do?

Quick nitpick, it's not the license that is the issue (emacs-elixir is using GPL), it is the copyright assignment to the FSF. Looking at the contributors list there are quite a few people, and I think it would be quite difficult to both contact them and get them to assign copyright to the FSF. So due to that difficulty my vote is to keep it separate and continue publishing to MELPA instead of ELPA.

How should we get more volunteers to step in and help maintain the project?

My main idea would be to make a few announcements asking for contributors, such as making a post on the ElixirForum and on Twitter. And additionally we could leave an open issue for a few months on this repo.

How should we document how to get an entire Elixir mode configured? Should we publish instructions like in this ElixirForum thread?

As long as we're not changing scope to be a "one-stop-shop" I think this is out of scope. Although linking to more comprehensive documentation would be great IMO.

Should we combine our efforts with what ElixirLS is doing? Should we shift to a "we expect ElixirLS"?

I'm generally in favor of keeping emacs-elixir separate from ElixirLS and instead aim for the two projects to be complementary (this is basically the same thinking that I posted above with respect to being a "one-stop-shop")

Do we want to support polymode for something like inline HTML templates

I think this is a good goal, but I'm not sure that much work is required to support this setup. Previously I was using polymode without too many issues (and none of them seemed specific to emacs-elixir)

[jsmestad]

Yeah good points on each item.

Now I totally appreciate this is the Emacs way and it is what you see from say using Ruby in Emacs. However when it was just alchemist.el it was WAY easier to get going with Elixir in Emacs and have a killer experience. Since the deprecation of that library things have become much more involved to get an experience anywhere near that of other editors for Elixir.

The pros / cons with the "one stop shop" idea to me is to follow more grouped convention (like alchemist.el did or a grouped package like evil-collection). I would imagine this looks like having an exunit.el, mix.el, and elixir-mode.el. The advantage is the documentation can speak to each of them as a holistic solution to getting Elixir running in Emacs. When it's a-la-carte, people struggle to get things working together as each project only documents it's responsibilities and not the interop between them. This holistic documentation has often come out of projects like Spacemacs or Doom to show the broader community how the packages can be combined and mapped to work together.

It also suffers from a users ability to discover these auxiliary libraries.

I think evil-collection has an elegant answer to this. If I want to do whatever to evil, I add (use-package evil-collection) and then configure what parts I want and which that I do not. The benefit is everything is there if I need it, but decoupled to allow for being customized.

I think right now it's quite a bit of effort to get Emacs going with all the bells and whistles for Elixir. For example, I currently have to get the following figured out:

• elixir-emacs
• mix.el (to run mix tasks)
• lsp-mode.el (or eglot.el) (for code completion)
• alchemist.el (this still provides functionality lacking from elixir-emacs and other packages)
• exunit.el (if I want test runner)
• polymode.el (if I want web-mode to work with inline liveview templates)

---

I think there should be a way to get back to that great (use-package alchemist) experience and get people off and running again hopefully driving adoption of the language. This approach should not require us to sacrifice customization that we all came to Emacs for in the first place.

[andreyorst]

Here's my two cents. Note, I'm not writing in Elixir, only looking at it, so I can be seen as a customer that can be turned off by the language support. Sort of. Anyway, I mostly write in Lisp, and very rarely in things like shell script, Lua, Perl, C, Rust, and some other languages. So I would like to look at elixir-mode from the perspective of languages that have great Emacs packages.

In my opinion, when user installs a major mode in Emacs, this mode should have essential functionality to use the language without external tooling. For example, Racket mode. This turns Emacs into pretty much complete Racket IDE - not only it provides syntax highlighting and indentation support, but it also has a REPL, completions, and a step debugger. Without such mode, users would be locked to use Geiser, which does not have a debugger, for example, so this feature is leaved out completely. Another example is Clojure, and while its major mode provides syntax highlighting and indentation handling, it also has some refactoring tools, that are meaningful for the language, and a very basic inferior clojure mode REPL. So again, it can mostly be used as a single package to get most of the work done, but more programmers use it paired with CIDER, thus making Emacs is a very capable IDE for Clojure. Same with common lisp and SLIME.

But these are Lisp languages that are known to have great support in Emacs, so what about other languages? Rust has a very basic major mode, which only provides syntax highlighting and indentation, but it also has ability to compile and run code, run Rust's linter, and code formatting. And paired with LSP support Emacs becomes a very capable IDE for Rust. This story repeats for pretty much any other language I've listed, because major modes for most of those were developed when LSP either did not exist, or was not yet adopted in Emacs.

Elixir kinda different in this regard, as it had Alchemist, which provided tons of features, but it was abandoned in favor of LSP, which actually can do less things, as it is not specific to Emacs. This gave birth to mix.el and other tools that were kinda extracted from Alchemist because LSP does not feature way to run tests for example, which can be done with Mix.

So in my opinion, major mode for Elixir should provide the following features that should not depend on any external tooling:

• syntax highlighting,
• indentation support,
• inferior IEx mode, so code can be evaluated from the buffer (kinda requires external tool but it comes with a language).

And the following features, that do require external tools:

• code formatting (currently handled with mix.el)
• code compilation, running, task running (also in mix.el)
• test running (via exunit.el)

Also a good support needed in widely used packages, such as

• Flycheck - has support via Credo
• Company mode - maybe a custom backend that can use IEx for completion? I'm aware that there's some package that runs IEx in background for completion to work, though I can't find the package, but maybe this can be a possible target?

LSP support makes last two integrations from above less needed, as it provides both completion and linting, but I think language still should be comfortable to use without depending on LSP for a bit of intelligence. Yes LSP is superior in some way, and it provides consistency, but this doesn't mean that this should be only option. Again, Clojure, there is a Clojure LSP, which evolves over time, but CIDER also is evolving and those two can actually be used together with a bit of configuration.

From my point of view Alchemist for Elixir was what CIDER is for Clojure - a well put package for interaction with a language. Which is actually a more or less unique feature of Emacs editor, to have a dedicated smart integration package for the language. Most other editors usually have generic support for each language, without too much smart stuff, and only recently LSP started to be widely used across editors to make common enchancements across those. Emacs always was different in that regard, for example it had semantic, Lisp modes had all this cool stuff around for years, which can directly compete with modern day IDEs. I believe that Elixir can be no exception here.

[Trevoke]

I like the direction andrey is going in, here.

I'm more conservative and I think the only things a major mode really needs to provide are:

• syntax highlighting
• indentation support

I'm not averse to more, but I think we should start with a conservative goal, given how much effort and expertise we have available right now.

I agree that the many packages, in the beginning, can be hard to manage for an end user, but they're infinitely easier to manage for open source maintainers, and we can always ask for unifying features if they're necessary. I much prefer a standalone mix.el package that other packages can safely depend on, for instance.

[victorolinasc]

First of all thanks to @jsmestad for this issue and the PRs and contributions made to this package. I am currently (last months) very hard at work and hadn't had the time to look into open-source more broadly unfortunately.

I agree with the general though here that a package that aims to implement a major mode shouldn't have more than syntax highlighting and indentation support. That is its main purpose and, IMHO, should be its only purpose. Both points might have a break-through in emacs devel with tree-sitter or something similar coming in the future. It is being actively discussed I think at least.

We could help the community have a "go to" package that would concentrate mix.el, iex.el, exunit.el, eex.el and all other goodies we can think of. Maybe have this centralized in a package so that it could all come bundled in might be a better experience. It is a bit contrary to the "do one thing only and do it well" but in this case it seems there are more successful examples of bundles of functionality than not.

[jsmestad]

@victorolinasc @Trevoke yeah maybe changing elixir-mode would be the wrong place to try and do things 'all in one'. I wonder if taking on reviving alchemist would be the better path forward.

[victorolinasc]

IMHO a better path would be to try an reach people responsible for mix.el, exunit.el and others to join on a new project where all of them would be maintainers together.

We could start by pinging them here for their opinion, WDYT?

At this point I believe it would be better to have a new project with this new aim that could bring the code from these smaller projects and build up a new one-stop-shop (plus the major mode) for this.

[jsmestad]

@victorolinasc I think for the new project I would aim for something as complete as alchemist was. That would mean the known extensions plus several more as there are still quite a few things missing

[victorolinasc]

@jsmestad that is what I thought too but their work could be used as a starting point IMHO. Unless there are very huge architectural changes or something like it.

I think that with more people onboard the chances of success might be higher.

[jsmestad]

Agreed

[axelson]

I'm also agreed on a new package tying everything together, and that would be as comprehensive as alchemist was.

Looking back at the original questions I think we just need to settle these two:

Can we update the README to detail out who the maintainers of the project are?

I propose adding victorolinasc, Trevoke, and myself (but please let me know if I'm missing anyone)

Should we publish a ROADMAP.md to show areas we think need to be added or revised?

I'm okay with adding a roadmap, but I'm not sure what would go on it

[viniciussbs]

Hi, guys. Any news about this topic? I would like to help, but I just started learning Emacs Lisp - even using Emacs for a long time.

IMHO a better path would be to try an reach people responsible for mix.el, exunit.el and others to join on a new project where all of them would be maintainers together.

I agree. mix.el already already offers some testing features like exunit.el, so maybe there's already some duplicated work happening.

[victorolinasc]

I think a bit has changed since we last discussed this here. IMHO we are almost at a point where the "purpose" of this project is not needed anymore... and this is actually a good thing :)

Currently, lsp-mode and similar gives us a very interesting IDE set of features with extensibility capabilities (actions). On the syntax parsing and coloring I think that the tree-sitter project is becoming the de-facto standard. Emacs seems inclined to implement this on the next big version and meanwhile the project https://emacs-tree-sitter.github.io/ seems well maintained.

Now that a tree-sitter parser is maintained by the core team, I think it will practically fulfill all the targets of this project. I think we could focus more on a better lsp experience and customizable actions as plugins with bindings into Emacs.

This would also make the devEx more standard across different editors. Of course, we are not there yet and so we might want to have everyone using a single package with everything.

[jsmestad]

I agree with @victorolinasc

[Trevoke]

I think the path ruby followed is one that we can follow too:

1. have elixir-mode be part of emacs: no dependency, which means kill the formatter logic in here
2. create enh-elixir-mode, a package that can be downloaded separately, which can do formatting, syntax highlighting, etc. using external dependencies, including lsp-mode if we so choose

[Sleepful]

Read through this and it leaves me wondering "what's the purpose of this mode?"
Seems like indentation and font locking are handled here, but someone mentions that tree-sitter could be better at this.
And the readme also mentions navigation, but it doesn't really tell me what it does for navigation at all and LSP usually handles this navigation unless you are adding features that go beyond LSP.

So, I wonder:

• to someone new to configuring elixir in emacs: Should they install this package at all?
• to someone with the desire to extend elixir IDE features in emacs: Where should they look to contribute if the idea doesn't fit the LSP protocol?

[victorolinasc]

@Sleepful I'll try to address some of these concerns from my point of view. This is just my opinion though.

First, we will always need a major-mode for elixir. Even with all the goodies that, luckily, are coming our way with Emacs 27, 28 and now 29, we cannot "plug" tree-sitter, lsp and other things without a major-mode, so I think we need a package yes. It could, in the future, be proposed upstream as @Trevoke mentioned.

Then there is the question about "what should a major-mode package do nowadays?". Again, luckily, very recently Eli (Emacs maintainer) asked for help trying and improving the tree-sitter feature aiming to merge this before 29 is out. There, he thinks a major-mode must do:

• font-lock
• indentation
• navigation (beginning-of-defun-function etc.)
• imenu
• which-func
• thing-at-point

And then he states:

We think that font-lock is the absolute minimum, followed by indentation.

There is even a major guide on how to start re-writing packages with tree-sitter in mind (and Emacs 29) here.

So in my opinion, for now, we should try and make a version of elixir major-mode that uses this new functionality.

For more customizations, we should have a different package and/or just hook into available core functionality. One of those will be LSP. It seems that for Emacs 30 a full client implementation of LSP eglot will be merged into core too. Add to that the recent announcement that Łukasz Samson , @axelson and others are enhancing the LSP server a lot, I think we will have a full IDE experience with Emacs practically out-of-the-box.

So, I think that currently, we still need this package as is. In the future, there would be a thinner package for font-lock and other Emacs infrastructure of major-modes and the rest of the focus should be on LSP servers. Everyone with any editor that has an LSP client will be able to profit from that.

This is my current opinion on the subject :)

[wkirschbaum]

I am happy to build out treesit support for elixir, started hacking on it last week and happy to see it through. I started hacking on this project since December, but never made significant enough progress to create a PR. I wasn't initially convince that treesit would be the way forward, but now it seems much more likely. My initial hacking was to get navigation working as this is my biggest issue working in elixir today. This is essentially the forward-sexp-function implementation, which SMIE struggles to cater for as Elixir has (call .. (do_block) (end)) not (function (block) (end)).

In the last week I got much further than I got in the last couple of months using SMIE.

If using treesit, I don't see much code in this project being useful for emacs 29 onwards. It would be great to see if we can add a elixir-mode to emacs for font-lock, navigation and indentation, all covered by treesit code which can replace the current implementation. Is this something we want to do? Given that this is probably the largest code change for the major mode for many years, it might be easier to treat it as a rewrite.

[victorolinasc]

This is certainly welcome @wkirschbaum !

I just think that we should maintain a version that is pure SMIE. Emacs has a huge set of possible installation options and runs in a variety of environments. We should always consider it will at some point not have the elixir support installed outside Emacs.

@wingyplus created an issue to discuss this topic and I think we all agree here that treesit is the way forward. So let's consider this as certain for the future of this project and discuss on #495 the implementation details.

Wdyt?

[wkirschbaum]

Maybe a quick update:

• elixir-ts-mode is now in emacs core for version 30
• lsp-mode, eglot, yasnippet-snippets ( and maybe some more ) all works with elixir-ts-mode out of the box
• treesit-auto should work with elixir-ts-mode now ( happy to expand on the changes if it does not )
• elixir and heex language servers sits in carouri's scripts: https://github.com/casouri/tree-sitter-module
• the purcell/setup-emacs@master github action ci thing works with treesitter now for 29
• there is a very experimental wkirschbaum/surface-ts-mode project ( waiting for comment )
• credo-language-server is going into lsp-mode ( hopefully soon )
• ....

Next:

• I think we need a super basic emacs cores elixir-mode, not ts-mode.. but don't think we use SMIE, it is too much of a time sink and does not really work well with Elixir.
• I am trying to work with Mitch Hanberg to get some of the new language servers to work with Emacs...
• elixir-ts-mode needs lots of tweaking, so feedback will be welcome.
  *heex-ts-mode needs to be improved. there is an open issue regarding this, but will take some time: Directive blocks are not represented phoenixframework/tree-sitter-heex#28
• I see neovim worked past this issue and think we should perhaps rethink how heex and elixir-ts-modes work together

[victorolinasc]

cc @jsmestad @Trevoke @axelson

So, I think the future is going in the following direction:

• elixir-mode and elixir-ts-mode will be part of core Emacs for Emacs 30 (a few years away maybe?)
  • until then, everybody on 29 will be advised to use the tree-sitter mode elixir-ts-mode which is available through package.el
  • everybody on 28 or less will be advised to use this package/mode
• the major modes should IMHO provide only basic syntax highlighting and simple in-buffer navigation
• all other extensions should better be in a language server for this to work properly cross-editors
• we could provide extra functionality in an extra package (in order to benefit everybody) with things like:
  • calling mix tasks (might also be done through language-server integration perhaps as code actions?)
  • specialized buffers for xref, eprof/fprof/compiling/tests/docs/etc (this may depend on running tasks)

I'd like to hear from others here about this broad view of the future :)