Shields as a Service

[ackerdev]

Let's discuss what we will need for the possibility of offering Shields as a Service. What considerations do we need to make to ensure that it is feasible and robust?

[ackerdev]

API interactivity
I was thinking of something like /travisci/ackerdev/myrepo/shield.png, which should in turn interact with a vendor's API to get the appropriate result to generate the dynamic PNG. This would require vendors to link through us for badge functionality, but removes the dependence of maintenance from the vendors.

Image generation
SVGs are awesome. It would be pretty easy to set up a base SVG to work with, and then manipulate it's nodes on the fly to quickly generate dynamic badges.
Problem: Converting SVGs to PNGs is a hassle. The way that I tried (and failed) at CodeClimate's shields was to use the 'rsvg' lib for imagemagick. I got as far as spitting out a converted version, but the external font absolutely mangled it. Could possibly be remedied by embedding the font into the SVG.
Alternatively, and this could be considered going the long way around the problem, I believe we could use Phantom to make a capture of the SVG (as long as it can come out with a transparent background). Don't know too much about Phantom, so would be great if anyone more familiar can weigh in.

[afeld]

I was just able to convert the badge SVG to a PNG with the following steps:

1. Download an arbitrary badge SVG file, e.g. rubygems_version.svg
2. Download the Open Sans font package
3. Download the imagick_type_gen script and save as imagick_type_gen.pl
4. Run the following:

chmod +x imagick_type_gen.pl
./imagick_type_gen.pl ./path/to/OpenSans-Regular.ttf > type.xml
# imagemagick doesn't seem to like the extra quotes around the font-family
sed -i.bak "s/'OpenSans'/Open Sans/g" rubygems_version.svg

# do the actual conversion
MAGICK_CONFIGURE_PATH=`pwd` convert rubygems_version.svg rubygems_version.png

open rubygems_version.png

Voila! Anyway, there's a relatively small number of possible shields, right? For everything but version numbers, it's only a handful. It might be easiest to pre-generate them and then have this service pull from a directory.

Another option would be to have a service that generates and caches them. I'd be happy to put something together with magickly (which uses imagemagick under the hood), if there's any interest. Should be pretty straightforward, actually.

[ackerdev]

Nice, though that still requires librsvg doesn't it?

Anyway, there's a relatively small number of possible shields, right?

CodeClimate has 41, Coveralls has just generated their 101, and GemFury's gem badge has a nigh-infinite amount.
It has to be a dynamic solution to be viable (though of course we can cache badges from vendors that are largely static, ala travis & gemnasium).

[afeld]

Not sure... do you know how to find that out? I just have imagemagick installed via homebrew.

$ convert -version
Version: ImageMagick 6.8.0-10 2012-12-16 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2013 ImageMagick Studio LLC
Features:  OpenCL

$ identify -list format
   Format  Module    Mode  Description
-------------------------------------------------------------------------------
...
      SVG  SVG       rw+   Scalable Vector Graphics (XML 2.7.8)
...

In any case, this sounds like a great use case for magickly :-) Will try to make a little sample service this weekend.

[ackerdev]

Hmm... that's right, ImageMagick comes with a default SVG renderer... I forget why I used librsvg. Maybe I'm misled because I thought it would have solved my font problem? I forget, it's been about a month since I originally tried to convert them with imagemagick.

Anyway, you would have to be installing imagemagick with the --with-rsvg (I think that's what it is) flag in homebrew to get librsvg (and a ton of the other deps, which is why I'm not too fond of it). You could brew search rsvg to see if you have it, but it doesn't look like you do.

[kookster]

I spent some time fiddling with this, and I get an image to render, but it doesn't look much like the png files:

vs

by adding -background to the imagemagick call, I at least get the corners to look right:

convert -background transparent template.svg result.png

I'm guessing that is not the right font, first off, even though I did the convert above, so fiddling with that now.

[kookster]

looking under the covers, it seems convert is formatting a call to inkscape for the actual svg->png.
I wonder if it might be easier to go straight to that?

> convert -verbose -background transparent template.svg result.png
"inkscape" "template.svg" --export-png="/var/tmp/magick-67103-okfqUBOeGP7" --export-dpi="90,90" --export-background="rgb(0%,0%,0%)" --export-background-opacity="0" > "/var/tmp/magick-67103jaAF40pZHoJx" 2>&1
mvg:/var/tmp/magick-671031U3Kq_F2Gzbh=>/var/tmp/magick-671031U3Kq_F2Gzbh MVG 77x19 77x19+0+0 16-bit sRGB 1.65KB 0.020u 0:00.009
template.svg SVG 77x19 77x19+0+0 16-bit sRGB 1.65KB 0.000u 0:00.000
template.svg=>result.png SVG 77x19 77x19+0+0 16-bit sRGB 0.000u 0:00.009

I was also wondering the default dpi; 90 it seems.

[cainus]

Hey all... I took a shot at this with limited success. See my "size" shield?

https://github.com/cainus/percolator#readme

I tried both imagemagick and graphicsmagick for the svg conversion but settled on graphicsmagick because the output was more correct than imagemagick.

These guys ( http://badge.fury.io/ ) get it right though:

https://d25lcipzij17d.cloudfront.net/badge.png?title=npm&type=3d&v=1.0.11test

So... Any clue how they're doing that? I guess in the worst-case scenario I could end up just drawing it programmatically without svg.

Thanks

[kookster]

I have had good luck installing inkscape, then using it from the command line.

Here is a copy of the travis 'build passing' image I generated from the command line:

Here was the command:
/Applications/Inkscape.app/Contents/Resources/bin/inkscape "travis_passing2.svg" --export-png="badge.png" --export-dpi="90,90" --export-background="rgb(0%,0%,0%)" --export-background-opacity="0"

I think this is workable for running on my mac at least so I can make the images I need, so I'm going to move forward with it. I suppose you could do the same on linux, but I haven't tried that yet.

[kookster]

I have been tweaking the text a bit more, using a transform on it 'transform="scale(1,0.93968341)"' to get the height better, and have now produced this from command line:

compared with this png from the repo:

This is really close I think, and now I am going to try and write up how to reproduce this result, and cleanup the template I am using for this. (I actually think the version I am getting from the command line looks a bit more readable, FWIW.)

What do you guys think?

[olivierlacan]

@kookster I actually like your render a bit better than ours. This is really good.

@cainus We could ask @rykov directly, that does look really good too.

I've been completely out of it for the last few months, I'm really sorry. Your experiments and any help getting something organized here would be much appreciated. @ackerdev and I bit off more than we could chew trying to build a gem before having a proper rendering process.

[olivierlacan]

@kookster Have you tried to produce a pixel doubled version? One of the reason the automated process is appealing is so that we can produce high density versions of all the badges very quickly if necessary.

[kookster]

@olivierlacan

No apologies needed. You guys gave an aesthetic blessing to all githubbers, but we all need to do other work eventually.

I did try out the render for retina by doubling the resolution specified in the inkscape command to --export-dpi="180,180" instead of `--export-dpi="90,90":

> /Applications/Inkscape.app/Contents/Resources/bin/inkscape "travis_passing2.svg" --export-png="badge_retina.png" --export-dpi="180,180" --export-background="rgb(0%,0%,0%)" --export-background-opacity="0"

Looking at this version, it makes me think the way I am doing shadows (putting a 50% opacity dark fill text slightly lower than the white text) is not quite right, but perhaps playing with the offset can make it look better. I'll comapre against your other retina images and see how close it is getting.

In any case, since the only change necessary was the command line, it at least answers whether the retina can be generated automatically in the same way.

[cainus]

I've got my node.js version running with inkscape (on linux) now at least: http://wapiti.io/api/sizeBadges/prozess . Inkscape renders quite a bit better than the other libs, so thanks for that tip. I think I just have to fix the square corners on the right side (probably a bug in my dynamic image sizing) and the drop shadow. I'd certainly be interested to see how you got that drop shadow.

Thanks again for the pointers.

[kookster]

Here is a quick version of what I did before I get embroiled in family holiday stuff (not a complaint ;).

I installed inkscape from inkscape.org

If you want to run it as a GUI, you have to tell it when it runs where to find X11 (on mountain lion I used /Applications/Utilities/XQuartz).

It is nice to be able to run it to make sure the fonts are loaded properly, preview changes, and see what manipulations to the svg Inkscape makes when you do things like change the height (this is how I figured out the values I use for the transform: scale(x,y)):
http://inkscape.org/doc/inkscape-man.html

I also needed to let Inkscape know how to find the Open Sans font.
I unzipped the set of font files into ~/Library/Fonts/Open Sans, then added a ~/.fonts.conf file like this (apparently this is how it knows where to find user fonts, as X11 loads them):

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<dir>~/Library/Fonts</dir>
<match target="font">
   <edit name="rgba" mode="assign"><const>rgb</const></edit>
</match>
</fontconfig>

After that I went into the Inkscape.app install to find the command line executable bin, which is here (on Mac obviously): /Applications/Inkscape.app/Contents/Resources/bin/inkscape

The command line I started with is from watching the verbose output of call ImageMagick convert. It apparently bundles some old version of Inkscape in order to do the svg -> png conversion, which is where I got the idea of using inkscape directly.

> convert -verbose -background transparent travis_passing.svg badge_im.png
"inkscape" "travis_passing.svg" --export-png="/var/tmp/magick-37749PRYjIrHJWq0f" --export-dpi="90,90" --export-background="rgb(0%,0%,0%)" --export-background-opacity="0" > "/var/tmp/magick-37749fhm4nS86L2K3" 2>&1
mvg:/var/tmp/magick-377493L8RJ0tK7BjE=>/var/tmp/magick-377493L8RJ0tK7BjE MVG 77x18 77x18+0+0 16-bit sRGB 1.71KB 0.010u 0:00.029
travis_passing.svg SVG 77x18 77x18+0+0 16-bit sRGB 1.71KB 0.000u 0:00.000
travis_passing.svg=>badge_im.png SVG 77x18 77x18+0+0 16-bit sRGB 0.010u 0:00.019

From this I used the following for standard size images:
/Applications/Inkscape.app/Contents/Resources/bin/inkscape "travis_passing2.svg" --export-png="badge.png" --export-dpi="90,90" --export-background="rgb(0%,0%,0%)" --export-background-opacity="0"

and this for retina:
/Applications/Inkscape.app/Contents/Resources/bin/inkscape "travis_passing2.svg" --export-png="badge.png" --export-dpi="180,180" --export-background="rgb(0%,0%,0%)" --export-background-opacity="0"

Then I used a variety of svgs, and they all seemed to work so long as the text font family was updated to Open Sans. I did find when editing in Inkscape that I needed to save as an Inkscape svg, not a Plain svg, but the extra meta-data it added does not seem terrible, and it seemed to render the existing svgs from the shields project without issue.

I want to clean up the svg before I promote it, but here is my latest version:
https://gist.github.com/kookster/5652987

A few tricks -

• I increased the opacity value of the shadow to 50%, and sneakily used a dark green color for the shadow on the green text (it felt sneaky anyway). This green may be unnecessary.
• I have fiddled alot with the y offset of the shadow text, not sure I have found the 'magic' value for this.
• I added a second identical white text element directly on top of the first. This made the white really pop, and much more closely match the existing png. No idea why this was needed, but it seriously helped.
• I tweaked the text in Inkscape GUI, and saw that it applied the following transform to make it the correct height: transform="scale(1.0,0.93968341)" It actually had a slightly higher than '1.0' x value, but it changed the width significantly, so I rounded back to 1.0, and it looks right.

Hopefully that is enough for anyone to get as far as I did, but I'll keep working to make this more systematic and templated.

[kookster]

@cainus I missed your message while I was typing mine.
I used one of the shields/travis/travis_passing.svg to start with, and that seems to have corners that work.
Maybe try the svg from my gist? https://gist.github.com/kookster/5652987

[cainus]

That was a huge help: http://wapiti.io/api/sizeBadges/Percolator and http://wapiti.io/api/scoreBadges/Percolator use the same dynamic svg generation code, so I have a working service now (with absolutely terrible code). It seems that the gradient markup from the travis svgs is much better than the gradient markup from https://github.com/olivierlacan/shields/blob/master/shield.svg .

I'd like to open source this at some point, but it's absolutely terrible. I took a bunch of shortcuts around the tougher problems (like computing text width), and there is no caching or memoization of anything so it would be a pretty terrible web service if used in production. I'd eventually like to just have a dead simple interface like

var shieldStream = shield(name, value, color);

Instead of the pile of assorted x's and y's I currently pass it.

Anyway... I'll keep improving this if no one comes up with anything better. I really appreciate the pointers so far.

ALSO: please let me know if it's still not looking quite right. I have almost no eye for detail whatsoever. ;)

[kookster]

@cainus - Those look good to me. I agree that a service will require more to scale. I think I will work on the ruby gem to be able to easily generate these, and a cleaned up svg. Then maybe a heroku build pack for Inkscape or something to turn this int a service.

Any tips on how you are calculating width based on text length would help. I figured I could try to to draw the text into an image by itself, and then measure it, or come up with some heuristic based on maximum char width.

Anyway, nice collaborating with you ;)

[kookster]

Making a heroku buildpack for inkscape sux, so it may not be usable for a heroku based service:
http://stackoverflow.com/questions/16803478/make-a-heroku-buildpack-for-inkscape-using-vulcan

However, in researching buildpacks, I found this svg2png-cli module, and it is a phantomjs based project which may make more sense for deploying, and it renders pretty nicely:

Here is the project:
https://github.com/skyzyx/svg2png-cli

Install:
npm install svg2png-cli

Here is an example image using the same svg as I used with inkscape:
svg2png -s 1 travis_passing2.svg

...and as a retina 2X:
svg2png -s 2 travis_passing2.svg

I'm going to pursue tweaking the svg to be more optimal for this phantomjs based renderer, and then work on a buildpack to turn this into a heroku hosted service.

@cainus, it may also appeal to you since you are already making an npm based app, yes?

[olivierlacan]

This is fucking georgeous @kookster. We need to start rolling out Retina badges with this.

[cainus]

@kookster: looks great! (Sorry about the slow reply... I was on vacation last week.)

Phantom is a good choice for two reasons too:
(1) it's probably easier (possible?) to run on heroku than inkscape
(2) you might actually be able to use the dom's getBBox() or text.getComputedTextLength() to solve the other problem of text with variable length.

I just didn't expect the output to be that good, and the api is entirely based on reading and writing files, so it looked a bit inconvenient. I'll take a crack at this too though when I get a chance.

[kookster]

you don't need to use the file based CLI, you can use the underlying
svg2png https://github.com/domenic/svg2png
agreed on the heroku ease of deploy, playing with that now.

Andrew Kuklewicz

On Mon, Jun 3, 2013 at 9:01 AM, Gregg Caines notifications@git.luolix.topwrote:

@kookster https://github.com/kookster: looks great! (Sorry about the
slow reply... I was on vacation last week.)

Phantom is a good choice for two reasons too:
(1) it's probably easier (possible?) to run on heroku than inkscape
(2) you might actually be able to use the dom's getBBox() or
text.getComputedTextLength() to solve the other problem of text with
variable length.

I just didn't expect the output to be that good, and the api is entirely
based on reading and writing files, so it looked a bit inconvenient. I'll
take a crack at this too though when I get a chance.

—
Reply to this email directly or view it on GitHubhttps://github.com/olivierlacan/shields/issues/15#issuecomment-18839270
.

[cainus]

@kookster: totally agree... but if you take a quick peek at that, it requires file-io (and therefore filenames in the api) to work as well. There's not really anything about the problem that should require disk access, and the disk access makes the solution a little trickier for a few reasons, if you want to run it in production. ANYWAY... It's probably most valuable to just get it running first. :) Lemme know how you do... I might get a chance to check into it myself tonight.

[olivierlacan]

Might not be relevant since our goal for now isn't to serve the SVG files but the PNG output instead but this tool looks like a great way to shave some weight off of SVG files.

[chadwhitacre]

Hey all, I messed up. I accidentally launched a badge-as-a-service app (http://badgr.co/) without reading this ticket first. I'm sorry. :-(

@olivierlacan and I met at Waza in February, so I was aware of Shields (and acknowledge it at the bottom of Badgr.co). I needed a fully dynamic badge for Gittip (see gratipay/gratipay.com#21) and I guess I shot first and asked questions later. Such is life. Sorry. :-(

Anyway, wanted to jump in here as soon as I became aware of this ticket. Looks like you're making good progress toward a solution here. How can I help? I would love to get Gittip in the mix as a badge vendor. I'm happy to contribute the badgr.co assets (domain, twitter handle, SSL cert, badger drawing :) ) if you're interested. Otherwise I'll just take it out back and shoot it when this gets launched. :-)

Again, my apologies. Keep up the good work! :-)

[ELLIOTTCABLE]

… and this is why I love @whit537.

[chadwhitacre]

Thanks for the heads-up, @ELLIOTTCABLE. :-)

[chadwhitacre]

@olivierlacan I've rewritten Badgr.co to use @cainus' work on shielded. Here's the pull request for this:

badges/badgr.co#16

And here's the new version (I'm thinking we'll put this behind a CDN, but for now the origin is fine for testing):

http://origin.badgr.co/

You'll see that I've also started rebranding it as Shields.io. I want the-service-formerly-known-as-Badgr.co to close this ticket. Can we talk about what we need to do to make that happen? Here's my proposal:

• land Here's a first pass at using shielded badgr.co#16
• start a clean repo with the work from the shielded branch of the badgr.co repo (there's a lot of old cruft that weighs it down, better to start fresh)
• call the new repo shields.io.
• redo the homepage and favicon on shields.io
• have a conversation about ongoing maintenance and funding. I'd like to fund Shields.io on Gittip, if possible.

I've staked out Shields_io on Twitter and Shields.io on Gittip. If you're open to it, @olivierlacan, I'm happy to include Shields.io within the Gittip org on GitHub and to run funding through Gittip, LLC.

How does this sound, @olivierlacan? Let's bring this in for a landing! :D

[olivierlacan]

All of this, amazing work. Let me know where to point shields.io

[chadwhitacre]

@olivierlacan Yay! :D

Let me know where to point shields.io

We'll need an ANAME/ALIAS to a MaxCDN URL. I'll let you know when I have that.

Do you think you and I could schedule another call? I'd like to hear your long-term plans for this project and talk to you about Gittip possibly "acquiring" Shields. :-)

[jbowes]

I implemented my own shields as a service project without checking in on this issue, and noticing there had been recent activity. oops!

The code is at https://github.com/jbowes/buckler with an instance running at http://b.repl.ca

Hopefully it can be of some use or inspiration. For example, I think the API it presents is a bit nicer, as the query string free URLs are more amiable to caching and are easier to type out by hand.

[cainus]

Ouuuch... Soooo much duplicated effort for something that makes $0.00 :) I'm guessing that you may have done a better job of it performance-wise though, because it looks like you've foregone svg entirely, if I'm not mistaken. It also looks like this can run on heroku, @olivierlacan , if you still want that performance boost: http://mmcgrana.github.io/2012/09/getting-started-with-go-on-heroku.html . Nice work, @jbowes !

[jbowes]

@cainus thanks! but yeah, ouch :(

Performance is pretty good. On my i7-3537u buckler will output a 12 char string png (as performance will be impacted by string size) in about 1.5ms (conversion to png takes up most of that time at about 1ms). So, performance is really bound by network.

Heroku should be no problem; I'm using OpenShift ATM with a Go cartridge that was based on a heroku Go pack:
https://github.com/smarterclayton/openshift-go-cart

[chadwhitacre]

Okay! I am happy to announce that Gittip has acquired the Shields project! :D

I'll write up a post about it for the Gittip blog, but here's the call where @olivierlacan and I did the deal:

http://www.youtube.com/watch?v=CZ0lgVWGSEs

And here's us shaking hands on it: :D

http://www.youtube.com/watch?v=CZ0lgVWGSEs#t=28m

On that call we transferred this repo from @olivierlacan to @gittip and renamed it to shields.io. The reason Gittip was interested in Shields is that Shields is a perfect example of a service that should be able to be funded on Gittip, and having one such service in house gives Gittip a chance to eat our own dogfood, drink our own whiskey, feel the pain, drive best practices, etc., etc.

Next steps:

• olivierlacan re-adds collaborators to the Shields.io team on the Gittip organization on GitHub: re-add collaborators #41.
• olivierlacan continues to manage the Shields project as part of the Gittip organization.
• whit537, et al. migrate the old badgr.co code into this repo: integrate badgr.co code into this repo #37.
• whit537, et al. move the issues on the badgr.co repo over to this one: migrate all issues from badgr.co over to here #38.
• Badgr.co has then become Shields.io!
• whit537, et al. put the finishing touches on Shields.io.
• olivierlacan hangs on to the shields.io domain for now (eventually we'll transfer it to Gittip, LLC: transfer shields.io to Gittip #39).
• olivierlacan configures DNS for shields.io per instructions from whit537.
• whit537 shares control of https://www.gittip.com/Shields.io/ with olivierlacan: give olivierlacan the reins to gittip.com/Shields.io #40.
• whit537 writes up a blog post for the Gittip blog.
• We announce this to the world! :-)

@jbowes I didn't notice your work until @olivierlacan brought it to my attention when we started our call. The performance is definitely a win over the Node version. Good work! :-) That said, the Node version works and is already integrated with Shields.io. We just shell out to it, though, so it shouldn't be too much work to swap out for the Go version. Can we keep this in our back pocket for when it comes time to further optimize Shields.io? Having buckler as a CLI instead of a web service would move us in the right direction.

Thank you and congratulations to @olivierlacan and everyone who has contributed to Shields so far. I'm excited to get http://shields.io/ online! 💃

[jbowes]

@whit537 certainly! cli access is on my todo list: badges/buckler#6

[chadwhitacre]

@olivierlacan I feel like we've gotten ahead of ourselves in our excitement. This is somewhat new territory so I don't think we should feel too bad, but at the same time we should try to set a good precedent here. The problem I'm seeing is that we didn't provide sufficient time for our respective communities to weigh in on this decision for Gittip to acquire Shields. We should have probably run those through tickets first before calling the deal final. As it is I've created a ticket on either side to collect any objections: #43 and gratipay/gratipay.com#1307. I propose that we proceed as planned but keep those open for a few days to give people a chance to weigh in.

[olivierlacan]

@whit537 Good call. I think we're more excited about getting shields.io than anything. 😉

[chadwhitacre]

@jbowes Awesome! :-)

[chadwhitacre]

@olivierlacan :-)

[olivierlacan]

💥 http://shields.io/

👏 @whit537

[chadwhitacre]

🙇

Huzzah! 💃