Setup ci/versioning/releases etc

[pandemicsyn]

Need:

• usual repo and branch protection rules
• precommit
• dependabot
• Semantic PR checks
• Issue templates.

[aaronsteers]

This would make a nice checklist or process document in handbook. I imagine we'll need to repeat this many times as we bring MeltanoLabs and other repos along with all the same best practices.

[pandemicsyn]

@aaronsteers Yea it probably wouldn't hurt if we setup a template repo - https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository . So at least future repo's don't require as much manual config.

[edgarrmondragon]

@kgpayne do you think this could be a good candidate for dynamic versioning?

cc @aaronsteers

[kgpayne]

@edgarrmondragon @WillDaSilva @aaronsteers

I think yes for the EDK, but I still have concerns about the copier template 🤔 Copier itself uses git tags to manage versions and to perform copier update actions. The docs state very clearly:

The Copier recommendation is: 1 template = 1 Git repository.

Why? Unlike almost all other templating engines, Copier supports smart project updates. For that, Copier needs to know in which version it was copied last time, and to which version you are evolving. Copier gets that information from Git tags. Git tags are shared across the whole Git repository. Using a repository to host multiple templates would lead to many corner case situations that we don't want to support.

So, in Copier, the subdirectory option is just there to let template owners separate templates metadata from template source code.

It may be safe to bump the version using poetry dynamic versioning (which also relies on git tags) but that would have some foreseeable issues:

• The versioning schemes may not be compatible/ideal. Copier creates tags in PEP440 form (e.g. 0.1.0) and we use semver in dynamic versioning elsewhere (e.g. v0.1.0). This is customisable in dynamic versioning, so may not be a blocker.
• We would be creating a new tag every time the EDK changes, even when the template itself hasn't changed. In the majority of releases therefore, copier would report a new version but users would receive no actual change-set.

I would recommend we create a new repo called copier-edk following the copier-template git topic to separate the two concerns 🙂

[edgarrmondragon]

@kgpayne I think the benefits of having the copier template in the same repo outweigh the issues, since there's also issues with spinning out the template, especially keeping it in-sync.

I took a look at the git topic and one of the first projects there is https://github.com/pawamoy/copier-poetry with dependencies extremely out-of-date, so I wonder if the git topic is how most folks discover copier templates.

[aaronsteers]

My preference is also to keep the templates together in the same repo with the EDK. This probably means that dynamic versioning is not an immediate option, and instead we'll need to use a version bump workflow like we have for the SDK.

While I understand that this also creates difficulties in referencing the templates from non-root paths, our precious findings on this front were that there were workarounds on our side that do not degrade the user experience.

@kgpayne - If it's worthwhile to dive deeper into other tradeoffs or implementation difficulties with the shared-repo approach, can you spin off a discussion for us to compile our thoughts around the respective challenges? Thanks!

[kgpayne]

@aaronsteers do you have a link to "previous findings on this front"? I can't find a working example of the approach you describe, and according to the copier docs what we are trying to do here in the EDK is explicitly called out as unsupported. Happy to spin off a discussion if there isn't a working reference or existing thread 👍

For context, I don't believe using a different release process (version bump) resolves the issue - version bump still creates git tags, which copier will interpret as new copier templates whether there are changes or not. It still means 'empty' copier updates for a majority of EDK releases. I understand the convenience of having templates with their libraries, but that isn't a use case copier supports yet. It might be possible to make it work, by adopting PEP440 versioning for the EDK and putting up with empty copier updates, but at the risk of compromising the copier update feature in some future release.

It may be worth experimenting with an example repo to see exactly how dynamic versioning and copier update could work together.