Wrong Dependency Repository Link

[mikepenz]

Using the new Dependency Submission API for maven packages links to a wrong repository.

Found https://github.com/mikepenz/gradle-dependency-submission/network/dependencies
The original repository would be here: https://github.com/Kotlin/kotlinx.coroutines

Looking up this dependency on Maven (and it's pom.xml file) shows the right repository link: https://search.maven.org/artifact/org.jetbrains.kotlinx/kotlinx-coroutines-core/1.6.3/pom

Is there more information available how this mapping is created?
I found some repositories on GitHub which map correctly to their repository. For example OkHttp, and Okio seen here

Not sure if it is related, but for OkHttp it shows Dependents, but I can't find any information how this is done. Is it a manual claim process so a maven package gets linked with the respective repository?

[laughedelic]

I observed the same issue in a Scala repository (also submitting Maven dependencies): https://github.com/scalameta/metals/network/dependencies

• most of the dependencies are not linked at all
• org.scala-lang:scala-compiler is correctly linked to https://github.com/scala/scala
• org.scala-lang:scala-library is supposed to be linked to the same https://github.com/scala/scala repository, but is linked to an old fork instead: https://github.com/javagems/scala-library

Pom files for these artifacts contain correct scm.url information, e.g. https://search.maven.org/artifact/org.scala-lang/scala-library/2.13.8/pom

[larsgrefer]

I noticed the same problem here: https://github.com/freefair/gradle-plugins/network/dependencies

org.springframework.boot:spring-boot-starter-json is wrongly mapped to https://github.com/ling185831070/Spring-boot-demo instead of https://github.com/spring-projects/spring-boot

Aside from pkg:maven package URLs it even fails to correctly link to pkg:github/gradle/gradle@a641d1caebe02ea109a69ca5fc0f59c20bceb9de

[ckipp01]

Also noticing this same problem when submitting projects from the Mill build tool via https://github.com/ckipp01/mill-dependency-submission. You can see for example this being used in https://github.com/com-lihaoyi/mill/network/dependencies with many of the same issues listed above.

[lefou]

It's also worth mentioning, that many links don't link to projects at all. This is especially true for almost all Scala dependencies. At least some of them have correct Maven metadata in their published pom.xml files.

[courtneycl]

👋 Hi from the dependency graph team at GitHub -- just posting to say we see this and are digging into it. Thank you for the detailed comments!

[lefou]

Yeah, if Maven pom does not contain a usable scm entry and here is also no project url to use, it's probably better to not link at all.

[mikepenz]

Thanks @courtneycl

In the small sample set I have there are for example more okhttp dependencies still referring to wrong repositories.
Looking at the pom file for these (like mentioned above) would point to the right github repo: https://repo1.maven.org/maven2/com/squareup/okhttp3/logging-interceptor/5.0.0-alpha.10/logging-interceptor-5.0.0-alpha.10.pom

Also for the org.jetbrains.kotlinx:kotlinx-coroutines-* not all seem to be handled yet:

[ckipp01]

Just found another incorrect one.

bloop-config 1.5.5 lives in it's own repo at https://github.com/scalacenter/bloop-config and the pom file that you can see here links to the correct repository. Currently this is just linking to itself.

[courtneycl]

@ckipp01 fixed!

Coming back to your previous qs:

I would assume that the heuristic is simply looking at these entries in the pom files?

It looks at the pom files, but it also does other things, and sometimes the other things overwrite what we've detected from the pom files. It needs some rework for sure.

Cleared (they were wrong; the source isn't listed in Maven Central):

I'm also not fully sure what you mean here. For example, here is the jsr305 artifacts on Maven Central

Sorry, I meant a GitHub-hosted source isn't listed. We currently only link packages to repos on GitHub.

[ckipp01]

So I don't really want to keep pointing these out as it seems like an incredibly inefficient way to do this, but looking at the deps in Dotty, there are a ton of them that are incorrect. Just to show a few:

Seeing that this discussion was started 8 months ago, I'm more interested in any information you might have about a roadmap for better supporting various ecosystems on GitHub since it's been quite some time that the API has been open. For example even just ignoring the incorrect links even if you submit a maven dep with another build tool that's not Maven you still see weird messages like this:

Showing that it can't find it in my pom.xml even though it clearly says right above it that it's coming from a build.sbt. Again, super happy this feature exists and I'm really looking forward to it being more usable from something besides pure java dependencies using poms. Is there any sort of public roadmap?

Ah, I also just now am seeing https://github.com/orgs/community/discussions/43364. I would love to still do this if it's a possiblity.

[jhutchings1]

Appreciate all the feedback, folks! We need to do some major work on our package to repository mapping logic for some of these ecosystems. Please bear with us as we take some time to analyze and prioritize any needed changes.

[Tyler-Angell]

Thank you @jhutchings1 for looking into this, out of curiosity, are Security Advisories impacted by this? As a sample, Microsoft.ApplicationInsights.WorkerService in my repo is reporting the fork vijay10193/asp1 as the source which hasn't been updated in a little over 3 years from the actual publisher microsoft/ApplicationInsights-dotnet. If the actual source repo publishes an advisory will I be notified of it without action from vijay10193/asp1

[jhutchings1]

No. We publish security advisories based on the package name/ecosystem, so that process sidesteps the package-to-repo mapping altogether. The package-to-repo mapping is purely used as a UI treatment in the dependency graph UX and we don't use that for any security conclusions.

[laughedelic]

@jhutchings1 it's been a while since the last update. Are there any plans or news you could share with us?

[courtneycl]

Hi @laughedelic -- we made some changes to our Maven repo mappings a ways back so that we'll only show mappings over a certain confidence level. This decreases the chance of having wrong mappings (though, not to zero).

We are currently working on a bulk ingestion of metadata coming from ClearlyDefined -- this is primarily for licenses, but it also includes some repo mappings, and we'll be assessing whether we can use that as a source. Otherwise, we have time dedicated for mapping improvement efforts in the summer.

[laughedelic]

Thanks for the update! It's good to know that some work is happening or is planned.

One concern about using metadata from any external source like ClearlyDefined is that it won't work for private repositories. As people in the threads above have mentioned, the metadata from the published pom files is probably the best source.

Another possible approach could be extending the Dependency Submission API with the optional repository mapping. That would enable community tooling to solve the problem in the way that is best for each ecosystem.