feat: Improve project bootstrapping

[janbuchar]

This adds a unified crawlee/project_template template. The original playwright and beautifulsoup templates are kept for compatibility with older versions of the CLI.

The user is now prompted for package manager type (pip, poetry), crawler type, start URL and whether or not Apify integration should be set up.

• closes Automatically install dependencies after bootstrapping a project via CLI #317
• closes Add more options to Crawlee CLI for crawler creation #414 (http client selection is not implemented)
• closes Optionally include Apify integration in project bootstrapping #511
• closes CurlImpersonateHttpClient warning on Windows #495

TODO

• http client selection
• disable poetry option if it isn't installed
• rectify the pip-based setup
  1. manual dependency installation - no automatic installation, just dump requirements.txt and tell the user to handle it any way they want
  2. pip+venv - dump requirements.txt, make a virtualenv (.venv) using the current python interpreter, install requirements and tell user to activate it
  - should be disabled if venv module is not present it's stdlib
• test the whole thing on Windows (mainly the various package manager configurations)
• fix how cookiecutter.json is read (it is not present when installing via pip)

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[janbuchar]

okay, this could use some testing now... plus there's a bunch of stuff to consider

• the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?
• does this work on windows?
• isn't the creation dialog too long now? maybe some of the options don't need the prompt and just CLI flags are enough
• should we include http client selection?
• we should probably scream and shout if poetry is not installed... or we could just disable the option in that case

[vdusek]

Nice, just a few comments

[janbuchar]

Thanks for the review @vdusek! Do you have anything to add regarding those points for consideration?

[vdusek]

the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?

That is generally how pip works. I wouldn't add additional overhead here. Simply dumping the dependencies into requirements.txt should be enough. Managing the environment and installation should be left to the user - that's the standard approach with pip.

isn't the creation dialog too long now? maybe some of the options don't need the prompt and just CLI flags are enough

IMO the current setup is fine. Also, all options have default values.

should we include http client selection?

That would be a nice feature, but I wouldn't prioritize it at this moment. We have a guide covering that, which should be sufficient for now.

we should probably scream and shout if poetry is not installed... or we could just disable the option in that case

I would suggest throwing an error. If a user wants to use Poetry, the option should be conditionally on Poetry being installed.

[vdusek]

the Pip versions does not work

[janbuchar]

the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?

That is generally how pip works. I wouldn't add additional overhead here. Simply dumping the dependencies into requirements.txt should be enough. Managing the environment and installation should be left to the user - that's the standard approach with pip.

I generally agree. But how would you ensure that pipx run crawlee create ... installs the dependencies in a venv? Often people will want to keep the virtualenv in the project directory, which doesn't get created until you run crawlee create.

should we include http client selection?

That would be a nice feature, but I wouldn't prioritize it at this moment. We have a guide covering that, which should be sufficient for now.

It's a 15-minute adventure, the only problem is that it'd make the dialog longer.

[janbuchar]

Relevant discussion - https://apify.slack.com/archives/C06ENBV1B9Q/p1728907202834259

[vdusek]

We provide http client option together with the PW crawler, is that what we want?

[janbuchar]

We provide http client option together with the PW crawler, is that what we want?

As long as we don't use playwright.request in PlaywrightCrawlingContext.send_request, it makes perfect sense. After that, it still might make sense in some niche cases, but the utility of giving that choice to users in project bootstrapping will be debatable.

[vdusek]

As long as we don't use playwright.request in PlaywrightCrawlingContext.send_request, it makes perfect sense. After that, it still might make sense in some niche cases, but the utility of giving that choice to users in project bootstrapping will be debatable.

I would not expose the option to the user, IMO it would results only to confusion. Net a request though.

[vdusek]

I tested a lot of variants, seems to be working, good job!