feat: parse requirements.txt with locally referenced packages

[madpah]

As per issue #284 raised by @Jonas-vdb.

Some development teams add local references to other packages in their requirements.txt files. This is currently causing an exception:

Traceback (most recent call last):
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 98, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/jvdb/repos/venv-tmp/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

[madpah]

Root Cause: pkg_resources again does not support parsing requirements of this format (similar to requirements with hashes included) - see here.

Also - worth stating that given the example provided, we will not be able to determine the version for any locally referenced requirements through a requirements.txt method. Other methods such as EnvironmentParser would pick them up.

[jkowalleck]

@madpah i dont see an actual urge for this feature, and i dont see it as a bug.
relative paths are not to be expected to be output of pip freeze

but it is a convenient feature, no doubt.

[madpah]

I agree @jkowalleck - will remove the bug label.

[mostafa]

@madpah
I was thinking about making a PR to replace pkg_resources with requirements-parser because the latter can parse all sorts of requirements. WDYT?

[madpah]

Hey @mostafa - thanks for getting involved.

This was absolutely my plan. However, since this ticket was created, the logic/classes that use pkg_resources now live in an upstream project cyclonedx-python - so I'll first relocate this issue (back) there.

Would be great to get a PR from you to make this change, but worth nothing we have a large(ish) release due to land probably next week now, so you might want to wait until that is merged before you branch.

[madpah]

This issue was moved back to cyclonedx-python as since version 2.0.0 of cyclonedx-python-lib, Python specific parsers now reside in cyclonedx-python project.

[mostafa]

Perfect! Just give me a nod when you release next week. Alternatively, I can start working on the feature off the branch you want to merge and then rebase from the default branch after merging. Up to you! 🙂

[madpah]

Hey @mostafa - we've released cyclonedx-python 3.0.0 today. Would be awesome to get a PR to move to requirements-parser as you suggested.

Thanks!

[mostafa]

@madpah I forked the master branch and tried to test the library locally but ran into an issue:
The tests fail because the method signature of BaseParser.get_components returns self._components which is a List[Component], but the code expects a set, so it can add to it; a method that is not supported on a list. There are 4 instances of this issue in various places. The following is an example of the tests that fail because of this error:

======================================================================
ERROR: test_conda_list_explicit_md5 (tests.test_parser_conda.TestCondaParser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/somewhere/cyclonedx-python/tests/test_parser_conda.py", line 46, in test_conda_list_explicit_md5
    parser = CondaListExplicitParser(conda_data=conda_list_ouptut_fh.read())
  File "/somewhere/cyclonedx-python/cyclonedx_py/parser/conda.py", line 41, in __init__
    self._conda_packages_to_components()
  File "/somewhere/cyclonedx-python/cyclonedx_py/parser/conda.py", line 68, in _conda_packages_to_components
    c.external_references.add(ExternalReference(
AttributeError: 'list' object has no attribute 'add'

Is this something you're aware of? Or should I create a separate issue?

[madpah]

@mostafa - not aware of this issue, and concerned as tests are passing in CI too. If you can raise an Issue for this, I’ll investigate later today.

Thanks again 🙏

[madpah]

Tests passing here: https://github.com/CycloneDX/cyclonedx-python/runs/5277735507?check_suite_focus=true

[mostafa]

@madpah I saw the passing tests, which is why I am confused, but I'll create a separate issue for it.

[mostafa]

@madpah It's a dependency issue on my machine. So, my bad. 🤦

[llamahunter]

We don't have locally referenced packages, but we do have a private pypi repo, and it appears that the pkg_resources library simply isn't up to the task of parsing a full pip requirements.txt file, including the --index-url directives.
#318

[jkowalleck]

Tool might not supports all the features people use in their requirements.txt.
the capability is a file that is created by pip freeze. @llamahunter

please open another issue, if you want to request another feature for the requirements parser.
this issue is about local packages.

[mostafa]

@madpah @jkowalleck
I had some issues with a test (test_example_multiline_with_comments) not passing after using the requirements-parser project because it doesn't support either the hashes or multiline requirements (either or both) and I found that there's a more mature alternative, the pip-requirements-parser. WDYT? As you're the maintainer/developer, do you strictly want to use and develop the requirements-parser package?

Also, the pip-requirements-parser package has only one API, which is RequirementsFile.from_file, and if we want to parse requirements file as a string, we need to make our own API using internal APIs of the package or get rid of our string processing altogether.

[jkowalleck]

just for the record: I do not have any opinion about a particular requirements-file-parser. Unlike @madpah I never dug deep into this topic.

---

regarding

Also, the pip-requirements-parser package has only one API, which is RequirementsFile.from_file, and if we want to parse requirements file as a string, we need to make our own API using internal APIs of the package or get rid of our string processing altogether.

For string input we could create a temp file and run RequirementsFile.parse/RequirementsFile.from_file on the temp file with include_nested=False, as all relative paths would be bare assumptions out of thin air based on current working dir and stuff.
For direct file input we would set include_nested=true, as relative paths from that file could be resolved.

[llamahunter]

please open another issue, if you want to request another feature for the requirements parser.
this issue is about local packages.

I already did. See my post above and #318

[jkowalleck]

for discussions, feel free to switch to #319

[mostafa]

One more thing that's definitely off-topic:
I see lots of <file>.close() lines throughout the project inside context managers, which is redundant because the file itself is actually being opened inside the body of a with statement that is a context manager and will close the file automatically, even if an exception occurs inside the body.