Libraries.io dependencies badge is flaky

[mosherc]

Are you experiencing an issue with...

• shields.io
• My own instance
• badge-maker NPM package

🪲 Description

Whenever I use the Libraries.io dependency status badge, results are inconsistent, and I get "invalid" as a status for most packages I try. Sometimes they will work until I add a query param (logo or badge style for ex.), then they will say invalid. Also it seems like the info is straight up wrong sometimes?

TypeScript example:

🔗 Link to the badge

https://img.shields.io/librariesio/release/npm/@ebth/design-system
https://img.shields.io/librariesio/release/npm/@ebth/design-system?style=for-the-badge

For me, those two are showing different results too, the link shows the correct number but the badge above shows invalid. Then when I add the query parameter, it is invalid. Also this keeps changing every time I reload the page or paste it somewhere else...screenshot of what I'm seeing in this issue below shows the inconsistency:

libraries.io entry: https://libraries.io/npm/@ebth%2Fdesign-system

As you can see, there are many dependencies that should be out of date. The Sourcerank endpoint also does not work for me.

💡 Possible Solution

I'm guessing that zero dependencies out of date is equivalent to "invalid". Would be nice if this just said zero instead? Many common packages are showing invalid.

side note
I know you can't get Github info for private repos, but it would be nice if that info (stars, number of releases, latest release date, etc.) from npm based endpoints like Libraries.io since those can be public with a private repo.

[calebcartwright]

Thanks for reaching out!

I'm guessing that zero dependencies out of date is equivalent to "invalid". Would be nice if this just said zero instead? Many common packages are showing invalid.

Nope, those are actually very different things. Unfortunately, the Libraries.io API has been notoriously unstable over a fairly long period of time, and at least as of the time of this post, is returning an HTTP 429 response code which is driving the invalid badges

[calebcartwright]

The behavior is indicative of rate limiting, but I also get the same result from my local so have added the upstream help label. I don't think we're using any api token for our badge server calls to the Libraries API which we could also investigate to see if it helps

[chris48s]

This works locally for me. http://localhost:8080/librariesio/release/npm/@ebth/design-system gives me

@calebcartwright if you can reproduce the error locally, what does npm run badge https://img.shields.io/librariesio/release/npm/@ebth/design-system spit out? Can you paste-dump?

[calebcartwright]

The failures are recurrent but not 100% consistent, i'm getting 429 at the moment but the badges in prod seem to be rendering at the moment.

Subset of the output

 Request  🏹 
 https://libraries.io/api/npm/%40ebth%2Fdesign-system/latest/dependencies
{
  "headers": {
    "Accept": "application/json"
  }
}
 Response status code  🎯 
 429
 Handled error  🙅 
 Error: Got status code 429 (expected 200)

    headers: {
      'content-type': 'text/plain',
      'retry-after': '60',
      'cache-control': 'no-cache',
      'x-request-id': 'ebd11249-89d3-4638-9c45-c919b1a63054',
      'x-runtime': '0.001401',
      'x-cloud-trace-context': 'bcb294d24f9c42caad14b7cfbc393e5f/14773168363079047931;o=0',
      'strict-transport-security': 'max-age=31536000; includeSubDomains',
      vary: 'Origin',
      'transfer-encoding': 'chunked',
      date: 'Sat, 13 Feb 2021 16:39:58 GMT',
      via: '1.1 google',
      'alt-svc': 'clear',
      connection: 'close'
    },

[chris48s]

Cheers 👍 This is all coming back to me now: 31c253f (this is actually the bower service, but same principle applies)

We are using the API anonymously because it was giving us a higher rate limit than authenticating 😆 If I just chuck a few requests at the libraries.io API it seems like they are now actually enforcing a rate limit on anonymous API requests.

As a first step, lets switch to calling them with an API key to increase our rate limit. I don't know if their documented limit for authenticated requests (60 /minute) is enough to totally fix this for the amount of requests we make. 60/minute gets us 3,600 per hour and according to the dashboard we served 1,680 libraries.io badges in the last hour and a few hundred on bower (which also uses libraries.io) so it'll probably cover us for non-peak times but I wouldn't be surprised if we exceed the limit at peak times.

[chris48s]

Unfortunately I haven't had time to work on this yet, but if anyone is interested in picking it up, here's some notes that scope out the job:

• Libraries.io uses query string auth ?api_key=
• AuthHelper has a withQueryStringAuth method to help us with this. Another service like https://github.com/badges/shields/blob/master/services/wheelmap/wheelmap.service.js or https://github.com/badges/shields/blob/master/services/twitch/twitch-base.js would probably be a good example to crib from
• There are 2 service families that use the libraries.io API - there's 4 libraries.io badges https://github.com/badges/shields/tree/master/services/librariesio and 2 bower badges https://github.com/badges/shields/tree/master/services/bower which would need to be updated.
• As well as updating the service code we'd need to add the config var to the private schema in core/server/server.js and config/custom-environment-variables.yml
• Docs in doc/server-secrets.md and doc/production-hosting.md should be updated
• I'm happy to donate my libraries.io API key for us to hammer in production. Given we will be quite tight on rate limits, we should ideally use a different key for integration tests. Another thing we could do if someone else is willing to donate a key, we could set 2 keys: 1 for bower badges and 1 for libraries.io badges (even though they are the same API) to try and stay under the rate limit.

[calebcartwright]

Another thing we could do if someone else is willing to donate a key, we could set 2 keys: 1 for bower badges and 1 for libraries.io badges (even though they are the same API) to try and stay under the rate limit.

I don't have an account but assume we could set one up (maybe using one of the shared shields.io emails?) and/or i could create one and donate one as well