Analyzer with SVN

[fb33]

Hi,

I've got an issue with a project under SVN.
I've got a project with this repo url : svn+ssh://svn.company.com/svnroot/Products/my-project/trunk
where there is a DotNet project : Prj/Src/module/module.vcxproj (ie => Products/my-project/trunk/Prj/Src/module/module.vcxproj)
The analyzer seems to compute a wrong vcs_processed url :

    projects:
    - id: "DotNet::Prj/Src/module/module.vcxproj:"
      definition_file_path: "Prj/Src/module/module.vcxproj"
      declared_licenses: []
      declared_licenses_processed: {}
      vcs:
        type: ""
        url: ""
        revision: ""
        path: ""
      vcs_processed:
        type: "Subversion"
        url: "svn+ssh://svn.company.com/svnroot/Products/my-project/trunk/Prj/Src/module"
        revision: "110884"
        path: "Prj/Src/module"
      homepage_url: ""
      scopes: []

the obtained vcs_processed url contains the path concatenated with.
So that makes the scanner failed :

Using scanner 'ScanCode' version 30.1.0.
Exception in thread "main" java.lang.IllegalArgumentException: The VcsInfo(type=Subversion, url=svn+ssh://svn.company.com/svnroot/Products/my-project/trunk/Prj/Src/module, revision=110884, path=Prj/Src/module) of project 'DotNet::Prj/Src/module/module.vcxproj:' cannot be found in Repository(vcs=VcsInfo(type=Subversion, url=svn+ssh://svn.company.com/svnroot/Products/my-project/trunk, revision=110884, path=), vcsProcessed=VcsInfo(type=Subversion, url=svn+ssh://svn.company.com/svnroot/Products/my-project/trunk, revision=110884, path=), nestedRepositories={}, config=RepositoryConfiguration( <SKIP too long config> ))).
	at org.ossreviewtoolkit.model.OrtResult.getFilePathRelativeToAnalyzerRoot(OrtResult.kt:279)
	at org.ossreviewtoolkit.model.OrtResult.getDefinitionFilePathRelativeToAnalyzerRoot(OrtResult.kt:265)
	at org.ossreviewtoolkit.model.config.Excludes.findPathExcludes(Excludes.kt:52)
	at org.ossreviewtoolkit.model.OrtResult$projects$2.invoke(OrtResult.kt:116)
	at org.ossreviewtoolkit.model.OrtResult$projects$2.invoke(OrtResult.kt:109)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at org.ossreviewtoolkit.model.OrtResult.getProjects(OrtResult.kt:109)
	at org.ossreviewtoolkit.model.OrtResult.getProject(OrtResult.kt:355)
	at org.ossreviewtoolkit.model.OrtResult.isProject(OrtResult.kt:493)
	at org.ossreviewtoolkit.model.OrtResult.isExcluded(OrtResult.kt:303)
	at org.ossreviewtoolkit.model.OrtResult.getProjects(OrtResult.kt:374)
	at org.ossreviewtoolkit.scanner.ScannerKt.scanOrtResult(Scanner.kt:97)
	at org.ossreviewtoolkit.cli.commands.ScannerCommand.run(ScannerCommand.kt:273)
	at org.ossreviewtoolkit.cli.commands.ScannerCommand.run(ScannerCommand.kt:207)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:395)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:392)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:410)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:435)
	at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:82)

If I store the project files on git ( my-project/trunk becomes my-project.git )
the analyser computes an other project definition, with a good vcs_processed url:

    projects:
    - id: "DotNet::Prj/Src/module/module.vcxproj:"
      definition_file_path: "Prj/Src/module/module.vcxproj"
      declared_licenses: []
      declared_licenses_processed: {}
      vcs:
        type: ""
        url: ""
        revision: ""
        path: ""
      vcs_processed:
        type: "Git"
        url: "ssh://gitlab/my-project.git"
        revision: "538d55b05fa178daa2ebfb5f10761d9e3aafacc2"
        path: "Prj/Src/module"
      homepage_url: ""
      scopes: []

And the scanner can work fine.

[fb33]

The PR #5303 seems to cause some regression in functional test, may be it's not the right way to fix this issue.

An other idea could be to change the way to fill the map the of relativePath (https://github.com/oss-review-toolkit/ort/blob/main/model/src/main/kotlin/OrtResult.kt#L259) by having a specific svn implementation of https://github.com/oss-review-toolkit/ort/blob/main/model/src/main/kotlin/Repository.kt#L71.
But I don't feel comfortable with that.

@sschuberth what do you think about ?

[sschuberth]

@Etsija, I believe your issue is a different one as you say

No analyzer-result.yml gets written.

whereas in @fb33's case the file does get written.

[sschuberth]

The PR #5303 seems to cause some regression in functional test, may be it's not the right way to fix this issue.

@fb33, it happened so that I'm looking at old Subversion stuff again... would you have / be able to create a test / reproducing repository for this issue?

[marcelp-px]

@sschuberth : Hi, is there a better workaround to accomplish the scanner step except to exclude all dependencies causing this error and include them manually in the report? These are now much more than 10 packages with this problem with the latest version 41.0.0.
e.g

private val excludedDependencies = setOf(
        "com.sun.xml.bind:jaxb-core:2.2.11",
        "com.sun.xml.bind:jaxb-impl:2.2.11",
        "xml-apis:xml-apis-ext:1.3.04",
        "xml-resolver:xml-resolver:1.2",
        "org.tallison:jmatio:1.5",
        "io.opencensus:opencensus-api:0.31.1",
        "io.opencensus:opencensus-contrib-http-util:0.31.1",
        "com.mysema.commons:mysema-commons-lang:0.2.4",
        "org.plutext:jaxb-svg11:11.4.0",
        "com.google.apis:google-api-services-translate:v2-rev20170525-2.0.0",
        "org.docx4j:docx4j-ImportXHTML:11.4.8",
        "org.docx4j.org.apache:xalan-interpretive:11.0.0",
        "org.docx4j.org.apache:xalan-serializer:11.0.0",
        "org.plutext:jaxb-xslfo:11.4.0",
        "com.thoughtworks.qdox:qdox:1.12",
        "org.springframework.security:spring-security-oauth2-authorization-server:1.3.3",
        "org.seleniumhq.selenium:selenium-ie-driver:4.19.1"
    )

[sschuberth]

@marcelp-px an easy work-around could be to configure ORT to prefer source artifacts over VCS locations for scanning. Of course that only works if the packages in question did publish source artifacts.

Other than that, if you'd like to help getting the issue fixed, ideally provide some public repository to analyze / scan where we can easily reproduce the issue with the latest ORT version (or sponsor the issue to get fixed 😉).