feat: use cargo generate instead of hardcoding examples source code

[paulotten]

Description of change

Use cargo-generate (as a library) to bootstrap copies of shuttle-examples apps.

Closes #867.

How Has This Been Tested (if applicable)?

Ran locally.

[paulotten]

@oddgrd what do you think of this approach using cargo generate?

It requires the user to have cargo generate installed, which I'll have to add documentation for.

In theory it's possible to include cargo-generate as a library instead. In practice I haven't been able to make the dependencies compile. (cargo-generate=0.18 requires git2=0.17, which conflicts with other things we use. cargo-generate=0.17 requires git2=0.16 which we use, but then fails to compile. cargo-generate=0.16 requires git2=0.14 which we is too low.)

[oddgrd]

Oh, cool, I didn't know about this crate! At first glance using that as a library seems like a great idea. That's a shame about the dependency conflicts though, I have ran into issues with the git2 dependency in the past (as you can see from the comment above it).

We would prefer to avoid needing cargo-generate to be installed, since that can lead to poor DX. We can install it for them, but that can also be finicky to get working smoothly across systems. 🤔 We don't want to reinvent the wheel either though... I think we should explore further if we can use it as a library (aka if we can resolve the dependency conflicts).

[oddgrd]

The dependency conflict should also be easier to resolve after we remove the cargo dependency, which is in progress. The cargo dependency currently ties down the git2 version.

[paulotten]

The dependency conflict should also be easier to resolve after we remove the cargo dependency, which is in progress. The cargo dependency currently ties down the git2 version.

I think I'm going to put this issue on ice until we've removed the cargo dependency.

[iulianbarbu]

@paulotten just out of curiosity: the cargo-shuttle dependency on cargo was removed here: https://github.com/shuttle-hq/shuttle/pull/765/files. Would you still be blocked by the cargo dependency that we have in the other places, when trying to fix the conflicts coming from git2 dependency?

[paulotten]

@paulotten just out of curiosity: the cargo-shuttle dependency on cargo was removed here: https://github.com/shuttle-hq/shuttle/pull/765/files. Would you still be blocked by the cargo dependency that we have in the other places, when trying to fix the conflicts coming from git2 dependency?

cargo is still a requirement in the top level (workspace) Cargo.toml. https://github.com/shuttle-hq/shuttle/blob/main/Cargo.toml#L44

That needs to go too.

[iulianbarbu]

Oh, that makes sense. Ok, thanks for the clarification! 🙏

[iulianbarbu]

#756 was resolved by #922 . @paulotten I am curious if we can continue with this PR. Do you have the time to take another stab at it? The idea is that we want to close #821 pretty soon and we're interested in getting this PR over the line. Let us know what you think

[iulianbarbu]

For some reason it doesn't work well on my side. Doing an init for an axum project brings over the entire shuttle-examples repo. Can you reproduce it?

[jonaro00]

I also get the entire repo

[jonaro00]

You can bump the examples submodule to the latest on main :)

[paulotten]

For some reason it doesn't work well on my side. Doing an init for an axum project brings over the entire shuttle-examples repo. Can you reproduce it?

Should be fixed now.

Now I'm noticing that the crate name isn't getting set correctly. Looking into it.

[paulotten]

You can bump the examples submodule to the latest on main :)

Fixed, I think.

[paulotten]

Now I'm noticing that the crate name isn't getting set correctly. Looking into it.

Should be fixed. (Ended up using toml-edit to set it.)

[paulotten]

Maybe just a tiny nitpick: the toml_edit imports could be at top of file.

Document and value feel pretty generic to be using everywhere in the file. That said we did have that before I started modifying this file. I'll move it back to the top.

I'm looking at if we can fix the failing rocket init tests and kill two birds with one stone.

Looks like just a new line at end of file difference. I'll leave it to you.

[jonaro00]

Looks like just a new line at end of file difference.

Yeah, but there are also some tests that fail sporadically. I think I've managed to reproduce it.

This can be merged if the team wants, and I'll add the fixes to the subsequent PR.

[chesedo]

Always love to see a PR that removes code.

Small nit: maybe the Shuttle.toml should be deleted since it will override the project to something that already exists.

[jonaro00]

@chesedo Yeah, either that or make it always create a Shuttle.toml. If many more features are planned for that file, then we might want it to be created in every project. (region, environments ... ???)

[paulotten]

Small nit: maybe the Shuttle.toml should be deleted since it will override the project to something that already exists.

Restating the problem to make sure I understand it:

Many (but not all) of the examples have a Shuttle.toml file. This file overrides the Shuttle project name to a value that is not useful.

So if we find a Shuttle.toml file after copying the example, we should delete it.

---

Ya, I can do that.

[paulotten]

Ya, I can do that.

There should be an update to this PR for this commit 65ab862

GitHub is having issues though. https://www.githubstatus.com/

Incident with Actions, Pull Requests and Webhooks

[paulotten]

GitHub is having issues though.

Appears to be fixed. I see the expected commit now.

[iulianbarbu]

LGTM! Thanks a lot for this. 🙏

[jonaro00]

I saw now that two things are missing! The init flag to generate does not initialize a git repo like cargo init does. One more thing that cargo does is add a .gitignore with target/. We should probably add this aswell but we can also include Secrets*.toml!

[oddgrd]

I saw now that two things are missing! The init flag to generate does not initialize a git repo like cargo init does. One more thing that cargo does is add a .gitignore with target/. We should probably add this aswell but we can also include Secrets*.toml!

Good catch! Should we include that here, or have you started on these in your new PR @jonaro00?

[jonaro00]

I have not started on those. You can do them here if you want.

[paulotten]

One more thing that cargo does is add a .gitignore with target/.

cargo_generate has a vcs option. Setting it to Git however doesn't seem to create a .gitignore.

We should probably add this aswell but we can also include Secrets*.toml!

cargo_generate isn't going to know to add that. I'll just write a .gitignore file manually.

[oddgrd]

One more thing that cargo does is add a .gitignore with target/.

cargo_generate has a vcs option. Setting it to Git however doesn't seem to create a .gitignore.

We should probably add this aswell but we can also include Secrets*.toml!

cargo_generate isn't going to know to add that. I'll just write a .gitignore file manually.

Thanks, sounds good!

[jonaro00]

Great.
The template could have a gitignore already. Should we append to the file if it exists? Or parse it and only add the lines if they are missing?

[paulotten]

The template could have a gitignore already. Should we append to the file if it exists? Or parse it and only add the lines if they are missing?

The simple version is if the template already has a .gitignore file, just use that that instead.

[oddgrd]

Good point! Another thought, can we use the git2 library for this?

[paulotten]

Good point! Another thought, can we use the git2 library for this?

It depends how complex we need to make this.

If we need something like this, I'd probably go reaching for a gix crate (Rust rewrite of git functionality). Maybe gix_ignore.

[jonaro00]

So shall we add a .gitignore with target/ and Secrets*.toml to all examples? It seems reasonable that they have this as a standard now that cloning will be the default way of starting off.

[jonaro00]

Setting it to Git however doesn't seem to create a .gitignore.

We can still pass that option so that a git repo is init'ed.

[oddgrd]

For git2 I was just thinking it would be nice if we could use that since we already depend on it, but if that gets complicated it's not worth it for this PR. I think we can do your suggestion Paul of generating the .gitignore if it doesn't exist, but I also think we can leave it out of this PR if we add a .gitignore to all examples as @jonaro00 suggested (which we should do either way).

[oddgrd]

LGTM, thanks for finding the cargo-generate solution and for finishing this up so quickly after the cargo removal! ❤️