Please consider maintaining the Emacs-Lisp files in a separate repository

[tarsius]

Current problem

This repository contains one or more Emacs-Lisp libraries, which are distributed on Melpa for easy installation using Emacs' official package manager (package.el) and are also mirrored on the Emacsmirror.

This project/repository is about much more than just the elisp libraries. These libraries are tucked away in some "contrib" or "utilities" directory. The maintainers of the project as a whole likely care little about these libraries, which makes it a less than optimal place to maintain them. The maintainers will always have something more important to deal with and potential contributors are likely scared away by the additional processes and conventions that such a large project needs, but which are overkill when looking at the elisp libraries in isolation.

Desired solution

I would like to propose that the Emacs-Lips libraries are maintained in a separate repository. Doing that would solve a few issues and have additional benefits:

Additional context

• Distributing a package that originates in such a large repository puts additional strain on the Melpa server. Fetching the repository takes longer and they require more space. We are already trying to deal with the latter, but not very successfully and this leads to special-cases in the tooling, which come with additional maintenance costs. See Strategy for huge repositories melpa/melpa#5361, also for some past discussion.

• I also maintain the Emacsmirror and here the costs are much higher than for Melpa. The Emacsmirror does not just distribute installable snapshots of a package, as Melpa does.

  Instead it mirrors the upstream repository. But in cases like this, where the repository contains much more than just the elisp libraries, it filters the history, using tools such as git filter-repo and git subtree. Unfortunately these tools are not very reliable and highly inefficient when continuously rewriting large repositories like this one.

  Currently this affects just 33 out of 8485 on the Emacsmirror, and the disproportionate amount of work that it takes to maintain the tooling for these special-cases is just not worth it. So I have decided to phase out support, but not before making an effort to get as many of these Emacs package slit out of the repositories they are currently being maintained in.

• For maintainers of this repository any issue about the elisp libraries is likely just more noise. They have better things to do.

• On the other hand, the original authors of the elisp libraries and people who have contributed to them, likely have notifications for this repository disabled because they only contribute to the elisp parts and there would just be too much noise otherwise. So issues and pull-requests by users are likely to go unnoticed for a long time.

  The author and contributors likely have to be explicitly summoned, as I am doing now:
  @jpkotta @adimasci @scop @ramiro050

• Potential contributors are discouraged when their pull-requests are being "ignored" or they might be discouraged by the additional requirements imposed by the project as a whole, and not even try to contribute as a result.

• Users who install elisp packages by manually cloning their repositories and then manually adding the appropriate directories to the load-path, are forced to clone a huge repository just to get their hands on a few elisp libraries in some subdirectory. (Or they can use the filtered repositories on the Emacsmirror, but as I mentioned, those are going away.)

So in summary, I think there are many downsides to maintaining the elisp libraries in this repository. I am not sure there are any upsides, except that it is the status quo.

Moving the elip libraries out of here is some work, that someone has to do, but most of that work is already done. You can just clone the respective repository from the Emacsmirror (at https://github.com/emacsmirror/pylint) and publish it somewhere else. (Please check that no recent history is missing and if the mirror repository is not uptodate, then ping me.)

Once you have done that tell me about it here, and I will take care of updating Melpa and the Emacsmirror.

Thanks for considering!
Jonas

[Pierre-Sassoulas]

I agree with everything you said. Among the current active maintainers we don't have ANY idea what to do when an emacs related issue is opened so we don't do anything. Another point to add is that we do not seem to have any emacs user (?), or at least no one read the documentation : https://pylint.pycqa.org/en/latest/user_guide/installation/ide_integration/flymake-emacs.html. We added that warning some time ago and had no issue opened about it since. I'd gladly accept a PR cleanup of emacs file + upgrade of the documentation and creation of another repository. I could even create it in pycqa and add the willing maintainers.

[tarsius]

I could even create it in pycqa and add the willing maintainers.

We can also add it to the Emacsorphanage, which I created for such situations. I would prefer if we could move this along quickly. Maintaining the Emacsmirror and related projects is a lot of work, and I have limited time for individual packages, especially if they fall into the "probably nobody uses this anymore" category.

So I would move the package from the mirror to the orphanage, update the mirror to get it from there, and update melpa to get it from the orphanage.

Should pylint/epylint.py be added to that repository? If so I would just do that using a new commit. Then I would look at the library header and clean it up if necessary. I would also create an issue which tracks the relevant issues from this repository.

At that point things are done as far as Melpa and the Emacsmirror are concerned, and things would remain as they are, until someone shows up and offers to take over as maintainer, or opens an actionable pull-request.

I'd gladly accept a PR cleanup of emacs file + upgrade of the documentation and creation of another repository.

My draft for the removal is at https://github.com/PyCQA/pylint/compare/main...tarsius:pylint:ex-emacs?expand=1, but I feel I am not the most qualified to this work, and I would prefer if you did it.

[Pierre-Sassoulas]

Sounds good !

but I feel I am not the most qualified to this work, and I would prefer if you did it.

No one is qualified 😄 I'm okay with nuking everything regarding emacs (except the doc specifying where the emacs utilities now lives). I'll have to think a little about epylint. Are you making design choice that not everyone agree with in the emacs world ? I.e. Do you know if someone is using epylint and not the solution you maintain ?

[tarsius]

No one is qualified smile I'm okay with nuking everything regarding emacs (except the doc specifying where the emacs utilities now lives).

Great, point at https://github.com/emacsorphanage/pylint as the new home.

I'll have to think a little about epylint. Are you making design choice that not everyone agree with in the emacs world ?

Have you ever read emacs-devel? Everyone disagrees with every thing.

I.e. Do you know if someone is using epylint and not the solution you maintain ?

I don't know. Does installing pylint also install "epylint", at least on some distributions? If so, then yes, we can only continuing to do that, by continuing to do that. On the other hand Melpa, or more generally package.el, is very established and how most people install most packages. (I don't, but doing this is also a win for people who do it like me, by cloning the repository, either manually or using an alternative package manager, centered around that idea.) So I don't think this is going to mess up anyone's way of living.

So one more thing I am going to do after adding epylint.py to the emacs-only repository, is to make sure the elisp code can actually find it.

[Pierre-Sassoulas]

Have you ever read emacs-devel? Everyone disagrees with every thing.

Actually laughed out loud at that.

So one more thing I am going to do after adding epylint.py to the emacs-only repository, is to make sure the elisp code can actually find it.

Sounds great, you seems more capable than me of dealing with angry emacs-devel dwellers.

Does installing pylint also install "epylint", at least on some distributions?

epylint is an entrypoint that is installed alongside pylint since forever. We could deprecate and see if someone complain. would https://github.com/emacsorphanage/pylint be able to package epylint in an epylint package ? (project name is actually free : https://pypi.org/search/?q=epylint)

[tarsius]

epylint is an entrypoint that is installed alongside pylint since forever.

If I understand correctly it is a wrapper around pylint that deals with the peculiarities of flymake. I.e., it serves no purpose if one isn't using flymake. So I would argue that it should live along-side pylint-flymake.el.

On the other hand changes to pylint might make it necessary to make changes to epylint and I wouldn't be qualified to make those. I propose that you get access to the new emacs repository, so you could make those changes. (I wouldn't absolutely expect you to do so pro-actively (after all, from your perspective, this is partially about getting rid of the emacs side of things), we could delay that work until an actual user reports breakage.)

We could deprecate and see if someone complain. would https://github.com/emacsorphanage/pylint be able to package epylint in an epylint package ? (project name is actually free : https://pypi.org/search/?q=epylint)

No. I should have read that part before writing the above. If you want to keep distributing this the way python tools are supposed to be distributed, then it has to remain in this repository. That would be absolutely fine with me too.

I have looked into how the elisp calls epyling, digging around in flymake, which I am not familiar with, or use. But things are clear enough. We would have to use an absolute path to the py file that lives next to the el file, instead of relying on it being somewhere on $PATH as was done before.

[Pierre-Sassoulas]

No. I should have read that part before writing the above. If you want to keep distributing this the way python tools are supposed to be distributed, then it has to remain in this repository. That would be absolutely fine with me too.

Removing epylint is a breaking change so we'll deprecate before removing for real that but other than that I don't mind if it's not distributed through python packaging, (I don't know how it's supposed to be used to be honest.)

I propose that you get access to the new emacs repository, so you could make those changes.

Sounds good, but yeah I'm not using emacs myself so it's best if someone using emacs push for it. I can try to answer pylint related question though and we're trying to fix what is causing problem downstream for all dependencies.

[ramiro050]

Thanks for working on this, @tarsius! The plan outlined seems perfectly reasonable to me.

[tarsius]

I've recreated the repository using git filter-repo --source /tmp/pylint --target /tmp/pylint-el "--path" "elisp/pylint.el" "--path" "elisp/pylint-flymake.el" "--path" "pylint/epylint.py" "--path-rename" "elisp/pylint.el:pylint.el" "--path-rename" "elisp/pylint-flymake.el:pylint-flymake.el" "--path-rename" "pylint/epylint.py:epylint.py" "--prune-empty=always" "--prune-degenerate=always" and made some small changes.

Then I published the result at https://github.com/emacsorphanage/pylint, and updated Melpa and the Emacsmirror to get it from there.

[Pierre-Sassoulas]

The result look pretty good, I like how you preserved the history selectively, that's neat 👍 This issue can be closed when we merge the pylint side's cleanup now, right ?

[tarsius]

I like how you preserved the history selectively, that's neat

For simple cases like this git filter-repo works pretty well, but don't try it on emacs' repository.

This issue can be closed when we merge the pylint side's cleanup now, right ?

Yes, but please ping me when you are done. Previously pylint-flymake.el assumed epylint was on $PATH. It still defaults to that behavior, but if that does not succeed it used the epylint.el from the directory same directory it lives in. Once you have unpublished epylint, it should do only the latter in case epylint.py is modified in its new home.

[tarsius]

I would like to create a tag, what should I use?

[Pierre-Sassoulas]

What would be the purpose of this tag ? Usually we only create tags for releases.

[tarsius]

What would be the purpose of this tag ? Usually we only create tags for releases.

A release tag is what I had in mind. 😄

Currently pylint.el contains ;; Version: 1.02, which likely just means nobody bothered to bump it.

The reason why I am asking what version string I should use is that the elisp repository also includes epylint.py, which was already released at https://pypi.org using some version string. The version used in the new repository should be either the same version or greater. Which should it be in your opinion? Also https://pypi.org/search/?q=epylint doesn't return anything anymore, so I cannot be sure what version was used before, without asking.

[Pierre-Sassoulas]

epylint is included in pylint, it was never released alone as far as I know. The last released version of pylint is 2.15.5. Next one 2.15.6, or 2.16.0, but most likely 2.15.6. We could create a package for epylint so it's independent, but you said it does not need to be python packaged so let's not.

[tarsius]

We could create a package for epylint so it's independent, but you said it does not need to be python packaged so let's not.

Agreed.

pylint.el is distributed on Melpa, which does rolling releases, which more or less means that it always publishes the latest release from the upstream repository using a timestamp as the "version" string. There also is a "stable" variant of Melpa, which distributes the latest tagged release.

I think it makes most sense to bump pylint.el from 1.02 to 1.3 and will do so tomorrow or so.

[nickdrozd]

FWIW, I use Emacs, and I don't use the Flymake integration. I just run Pylint in M-x compile and it works well enough.

[Pierre-Sassoulas]

Ho wow, that's it, we found a contributing emac's user 😍

[Pierre-Sassoulas]

@tarsius there's no issues tracker in the new repo, could you give me the right to administer it so I can fix it, please ?

[tarsius]

there's no issues tracker in the new repo

I don't know what happened there. I've added that now.

could you give me the right to administer it

I've send an invitation.

[silbe]

FWIW I only noticed this deprecation / pending removal because I happened to dive into the epylint source and saw the warning line. There's no output from epylint itself that it will stop working soon (despite the warning line).

I use epylint (rather than pylint directly) because that's what python-check in emacs invokes (I also add flake8 to the invocation but that doesn't matter for this discussion). Is there anything else I should be using instead?

The new repository is also very sparse on details; there's not even a README and the commit messages don't explain anything either .

[tarsius]

There's no output from epylint itself that it will stop working soon (despite the warning line).

Nobody is planning for it to stop working soon.

Previously the elisp lived in this repository but the maintainers didn't know how to deal with emacs related issues, so they didn't. Now the elisp lives in a repository being semi-maintained by people who know about emacs. So far no new issues were reported, but if something came up and had something to do with the elisp side of things, then we would be able and willing to deal with it. If a python related issue came up, then we might require more help from the user reporting the issue, but we expect they are capable of dealing with python issues, after all this implements linting for python.

I use epylint (rather than pylint directly) because that's what python-check in emacs invokes

(You are reporting that we don't provide enough information in a trivially consumable fashion, but here you leave out crucial details too. You don't mention how you install it.)

Likely nothing changes for you. Previously you got the package through melpa (I assume). Since melpa isn't about to drop the package, you can just continue to do so.

The new repository is also very sparse on details; there's not even a README

The current situation is described by the length discussion above. Yes, that is not ideal, but you have to understand that the pylint maintainers are not emacs users and that the emacsorphanage maintainer is not a python users.

and the commit messages don't explain anything either .

The commit message are adequate for what the commits do. There also is nothing in those commits that suggests that the functionality will soon evaporate. Quite the opposite, if you look closer, you will see that they are concerned with future-proofing -- making bitrot less likely, despite very limited planned maintenance.

[tarsius]

I've added a readme, which hopefully addresses your worries.