Automate badging review and issuance

[mattrosno]

I want to start the discussion on how we can automate the Developer Essentials badging process. I understand not everything would happen in this repository, but it's a good place to start.

Background

IBM Carbon Design System Developer Essentials badges are available based on the successful completion of Carbon tutorials.

The following badges are available to anybody who completes steps 1 through 5 of a Carbon tutorial, as demonstrated by an accepted pull request per step.

1. IBM Carbon Design System Developer Essentials - Angular
2. IBM Carbon Design System Developer Essentials - React
3. IBM Carbon Design System Developer Essentials - Vue

Each tutorial step illustrates different aspects of developing web applications with Carbon, from initial installation, to building pages, populating components with data, creating components and deploying to IBM Cloud.

The tutorials are valuable as education and activation material and I see opportunity for them to do more through additional tutorial steps. Prior to doing so, let's see what we can automate.

As is

Users create a pull request at the end of each tutorial step. Netlify deploy previews often run, but occasionally don't due to unknown reasons. There's a SurveyGizmo form at the end of each tutorial for users to apply for the badges. The form states "a few weeks" for badging turnaround time.

Then, representatives from the React, Angular, and Vue teams review and close tutorial step PRs weekly. Finally, I sift through SurveyGizmo responses on Fridays (but ends up being every couple/few Fridays) to look up PRs and if 5 are successfully commented and closed, I issue badges.

Proposal

We can automate both aspects of this: reviewing PRs and issuing badges.

Reviewing PRs

What if we seed each tutorial step branch with a failing test. The test doesn't need to be a full snapshot as that may be too strict, but some minimal test that would give us confidence that they correctly did the step. Successfully completing the tutorial step would result in that test passing. When creating the step PR, CI kicks in and runs the tests. If all tests pass, that PR gets a friendly comment, gets a label or something, and is closed.

Similarly, if the CI fails, there's a comment that something went wrong with encouragement to try again.

Would we need a GH bot for this? To receive new PR webhooks, watch a CircleCI build, and upon success, comment, label and close?

Issuing badges

Acclaim's API can issue badges. What if we hosted a form in Carbon website (or a different public-facing site if need be) where you choose a tutorial (e.g. React) and you input your GH username and hit submit?

In a serverless function (or something to avoid a server) we use GitHub's API to check for completed tutorial steps. If all looks good, we issue a badge through Acclaim's API.

Questions

Any other ideas how we could do this? Would we need a GitHub bot to watch the tutorial repo PRs to comment, label and close?

Can we avoid a serverless function to hit Acclaim's API? Since we need to protect that access token somehow, a serverless function would be easy especially on Netlify because it versions their lambda functions alongside side source. How much work would an IBM Cloud Function be for this?

[mattrosno]

@alisonjoseph @lee-chase @cal-smith you and the team reviewing tutorial PRs - what are your thoughts?

@joshblack any insight on the need for a GitHub bot for this? Do we need to host a GitHub bot for anything else where we could align this effort? (E.g. for CLAs, etc.)

[mattrosno]

@kellychurchill @andreancardona any thoughts on this with how the Enterprise Design Thinking badges are handled?

[cal-smith]

When creating the step PR, CI kicks in and runs the tests. If all tests pass, that PR gets a friendly comment, gets a label or something, and is closed.

Something like this seems perfect - as long as a full prod build passes, as well as a simple test suite, we can be reasonably sure that they correctly implemented the tutorial ... This is more or less my approach to reviewing anyway, people tend to stick pretty close to the code style and samples we provide, so the big thing is just making sure they didn't forget to do something.

Ideally we could even have the bot output to a slack channel (or 3) to give us a heads up when things are going well, or when someone is struggling

[mattrosno]

@cal-smith I like your idea of a feedback loop so we can be more proactive and can help guide when we now know things aren't going right for somebody.

[alisonjoseph]

100% on board with automating all of this. When reviewing I mainly check that the preview link looks correct and will glance at the changed files.

I like the feedback loop idea also. 👍

[mattrosno]

Updated proposal

The automated PR review is going great. Now we just need to provide a badge application form. As stated above in the as-is, that's currently a SurveyGizmo link that is manually reviewed. Right now somebody could fill it out and provide somebody else's PRs to game the system, but I haven't noticed that happening.

We need one thing to issue a badge: an email address. I don't know if we could find the primary email address associated with that GitHub user account, but if so, that could be an option. Also, each commit in each PR has an associated email address, but what happens if they switch their Git config mid-tutorial?

Both of those options won't work because most IBMers will want to use their IBM email address with the badge so it shows up in their BluePages account. It's not reasonable to tell people to change their Git config prior to their first commit just so tutorial step completion commits carry an IBM email address.

The best experience is to let people complete the tutorial with their default Git settings, and then specify what email address they want to use when applying for the badge. The best way to do this is to let them log in with GitHub and choose any email address that they have verified. Maybe it's personal, maybe they have verified their work email address in their GitHub account. Once they log in with GitHub OAuth, we can be certain that the form can't be gamed.

Because we likely don't want to add GitHub authentication to carbondesignsystem.com right now... I'm proposing we use the Gatsby theme to spin up a "micro-site" at badges.carbondesignsystem.com that has a log in page and the protected page that lets you apply for the badge.

We host it on Vercel. I have the necessary serverless functions for GitHub OAuth done here as a proof of concept: #1295. That proof of concept was just to get verified email addresses. We'd also need a couple more serverless function endpoints to check PR status and to apply for a badge using Acclaim's API.

I can do this so it's not requiring time from our dev team. I'd ask the design team to make a pass on the UI shell with auth, log in page, page to show available badges and status, and the form itself.

Thoughts @alisonjoseph @vpicone @tw15egan.

[vpicone]

@mattrosno I love the idea of splitting the login logic and badge api complexity off onto it’s own subdomain. I'm much less worried about getting the perfect implementation out of the gate considering we can iterate and improve on it separately from the carbon website.

I think the most automated, clean and cheat-proof method would be to use the organizationVerifiedDomainEmails query during the merge process. However, I don't think IBM has properly verified ibm.com, as that query isn't working for any of the users I've tried.

A work around for this is asking them to switch their profile's email to the one they want for badges. Our github action would query for the user's public email address, then send a request to Acclaim with that email. No forms/logins and we get automatic validation linking the user and the PR.

Just to reiterate, I'm down for whatever solution on its own subdomain, the method I described above just seems like the most hastle-free and bullet proof (no need to maintain a new repo, auth flow, or dns config).

[mattrosno]

Thanks @vpicone. I like the idea of keeping it to a GH action, and always a fan of not building more than we need to, but that's going to just cause more inbound noise for our team when people reach out with "I didn't update my public-facing GH email prior to completing my 5th tutorial step. Can you issue my badge?"

I also like keeping the badge application form separate from doing the tutorial because it doesn't force the badge on people. I know you can always not accept the badge when Acclaim emails you, but that may come off too forward to be issued a badge when that wasn't the intent, like a non-IBMer just looking to learn more about React and/or Carbon.

Let's roll with a sub-domain micro-site badges.carbondesignsystem.com. carbon-badges repo. Using Gatsby theme and Vercel functions so there are future portability options should this evolve into something else or in carbon-website natively.

I'll make it happen unless any objections. CC @joshblack @jnm2377 @alisonjoseph @tw15egan

[mattrosno]

DONE