Updated packaging proposition

[vaab]

Updated packaging proposition

The main objective here is to:

• move toward simplicity (although moving from single-file package to full fledged package)
• hit maximum compatibility with all versions of python packaging
• apply some standards

[tuukkamustonen]

Would you clarify why you like to stick with autogen.sh and d2to (deprecated, really)? I think they add complexity to build, and you don't see them in use anywhere else. What are the exact problems they solve?

For versioning, consider using https://github.com/pypa/setuptools_scm - it automatically reuses your git revision and is pretty nice.

MANIFEST.in is only required for source distributions - if you don't plan to release one, that one can be removed (wheels are the way to go).

[vaab]

There are 2 topic here: autogen.sh and d2to1.

For autogen.sh, well, I'm not ready (yet?) to change this part. autogen.sh is a very common solution (coming from autotools) that proved to be more robust that all the python packaging madness in the 10 last years. It has no requirements other than bash (which, I recon, might seem like a problem for windows compatibility, but any windows git installation has bash bundled with...). autogen.sh responsibility is clearly limited and defined : and I feel that this responsibility is code agnostic... allowing me to use the same autogen.sh with all my projects across all languages. I feel that this is clean. I understand some other people could feel differently about this, but if there are no clear reason to move away from this solution, I won't change it, as I feel it worked and prevented me to have to loose more time with anything related with the python packaging's moving target.
As you probably know it, autogen.sh tackles any configuration of the code to prepare a release : the generation of the changelog and changing the version info in various files. I'm aware of setuptools_scm existence. And actually used it on other project a long time ago and do not feel like using it : It was already a moving target 10 years ago and was poorly implemented judging from the number of time I had to spent debugging it (it wasn't supporting git BTW). I'm sure it is much better today, but I don't feel the need to move to a specific python solution for a task that is not specific to python.

As for d2to1, it still works and supports all the cases of the build matrix (namely working with old and modern versions of the python packaging tools in all versions, and that's more than a feat), maybe this won't be the case for long (especially because its name is based on distribute2 that is officialy deprecated, hence the rename to setup.cfg: https://github.com/embray/setup.cfg). Anyway it provided me somewhat a nice protection against a lot of inconsistency of python packaging across versions. And it still works. Not to mention that I feel much better having to use a non-executable file for metadata information (the very existence of d2to1, distribute2, setup.cfg, pbr, and the recent moves of setuptools itself, show that this point of view is quite shared amongst the python community).
On that subject, setuptools is moving towards the elimination of setup.py as it seems, and the last version are supporting setup.cfg configuration (pypa/setuptools#862 , doc: https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files) (albeit with slight variation with what d2to1 is providing).
Meanwhile, with d2to1 I remain compatible with a very wide versions of all the successive packaging tools and keep a nice and minimal setup.py that I can share amongst my python project.

I hope I managed to clarify my position on these topic, I don't expect you to agree on it...

But let's agree on the fact that this current PR is a step forward. Do you have any concerns on what it brings ? I'm looking forward to your comments if any and to push this as soon as possible: PR that change file location are a pain in the ass to keep open (as you know it, sorry for that).

[tuukkamustonen]

Alright, thanks for taking time to clarify the reasoning behind project tooling!

The reasoning sounds somewhat philosophical (e.g. that packaging should be described in config files rather than as code like with setup.py), rather than practical. And maybe a bit historical. But sure, if it works for you, why not. It's just more easier for people to contribute the more "standard" the project (packaging here) is. But then, you don't need to touch packaging very often anyway.

And yeah, I feel your pain - although I've been pretty happy with python packaging, I am aware of the "moving target" in python packaging world and I've read many agonizing stories by others :)

For changelog generation, release tasking etc. I've been using https://github.com/pyinvoke/invoke. I have collected all the common tasks in a library that I use from various projects (invoke's author has somewhat generic https://github.com/pyinvoke/invocations that he uses). It allows you to even avoid copying anything between the projects (although, copying single autogen.sh file is a no brainer). I recall running autogen.sh when I was playing with the project and it produced all sort of weird artifacts (to me) so I may have been a bit confused about what it actually does.

So, yeah, merge on by all means. Nice that #41 gets fixed. Good that you found time again to work on the library and thanks again for providing it for all of us!

[vaab]

Thanks for sharing your experience with pyinvoke, I did know about fabric so I see what is the target.