[python] handling multiple requirements.txt like files or custom pip-compile commands?

[adehad]

Package ecosystem
pip
Package manager version
pip 21.0.1
Language version
Python 3.8
Manifest location and content prior to update

1. https://github.com/pycontribs/jira/blob/master/requirements.txt
2. https://github.com/pycontribs/jira/blob/master/test-requirements.txt

dependabot.yml content
https://github.com/pycontribs/jira/blob/master/.github/dependabot.yml

Updated dependency
urllib
What you expected to see, versus what you actually saw
Of the two requirements files only one of them was updated.

We generate separate requirements for test and install using pip-compile:
https://github.com/pycontribs/jira/blob/f96495a6ca90b037c477761db3ad7a1cad0aaff0/tox.ini#L75

So it may be that we are using dependabot incorrectly for this setup.

Native package manager behavior
Not related to package resolution

Images of the diff or a link to the PR, issue or logs
pycontribs/jira#1064

🕹 Bonus points: Smallest manifest that reproduces the issue
I think have any two requirements files that specify the same package and dependabot would only update one

requirements.txt

urllib3==1.26.4

test-requirements.txt

urllib3==1.26.4

[JMMarchant]

We're also looking for this addition; we separate out into requirements.in, requirements-test.in and requirements-dev.in (each generating a .txt file and constrained by versions in the previous ones in the list). Being able to have dependabot recognise that and address multiple pip-compiled files would be a huge boon.

[whardier]

It looks like there is a pip-compile package ecosystem now that I will have to check out: https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates#package-ecosystem

Possible example: https://github.com/mozilla-releng/tooltool/blob/fb7d180f5c21286f443a709e57538643fd8b6707/.github/dependabot.yml

For reference in my own work:

I've had to start putting everything in setup.py with extras dev and types (more as needed) in order to allow for a useful dependabot workflow.

Ex:

pip-compile --extra=types --generate-hashes --no-emit-index-url --output-file=requirements.txt
pip-compile --allow-unsafe --extra=dev --extra=types --generate-hashes --no-emit-index-url --output-file=requirements-dev.txt

rather than using requirements.in... In situations where I am not providing a package (no setuptools rigging) I would LOVE to just use requirements*.in instead.

I feel like I'm missing an obvious preference here where dependabot should be able to be fed a list of requirements files (as requirements.in is fully compatible)

[adamjstewart]

+1, it is extremely common to separate required, optional, documentation, linting, testing, and developer dependencies into separate files. Some examples:

• cython has separate (doc|test)-requirements.txt files for each Python version
• numpy has separate (doc|linter|release|test)_requirements.txt files
• scipy has separate (doc|mypy)_requirements.txt files
• matplotlib has an entire directory of files for dev, doc, all, extra, flake8, and minver dependencies
• cartopy has an entire directory of files for required and test dependencies, as well as each extra

We would love to do this in TorchGeo too so we don't have to install all required/optional/doc/linting/testing deps just to run flake8 during a test.

[adamjstewart]

Actually, I just tried this and it seems to work? Both requirements.txt and requirements-min.txt were updated in this PR: https://github.com/microsoft/torchgeo/pull/603/files

Maybe this works fine but just isn't documented?

[adehad]

@adamjstewart is that the right PR reference?

[adamjstewart]

Ah, it looks like dependabot force-pushed and overwrote the original commit after I moved one of those files to a different directory. The original commit edited both files.

[deivid-rodriguez]

Yes, these days dependabot does not just trust the filename, but runs some checks to discover requirement.txt-like files. So this should be working now. Let me know if I missed something and this is actually not the case, so that we can reopen this! Thanks!

[pamelafox]

Could you update the docs to indicate that files besides requirements.txt will also be found? It wasn't clear until I dug around and found this thread.

Docs:
https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#pip-and-pip-compile

[JonathanRenon-EDB]

Hello @deivid-rodriguez,
Not sure if it's part of this issue, I'll gladly open a new one if needed.

we have a single requirements.in that generates 2 different .txt files one using --generate-hashes and the other one doesn't (one of the tool we need doesn't support hashes)

dependabot will only keep one of them updated, the non hashed one, the 1st in lexical order, not sure how it get chosen over the other, but it's constantly this one.

[deivid-rodriguez]

Hei @JonathanRenon-EDB! I'm no longer part of the Dependabot team. I recommend to open a separate issue explaining your current problem, though. This has been closed for too long without complaints, fresh issue is better 👍.