Payments API v2

[brianlovin]

This PR updates all of our payments logic to an entirely new system that lives on a new worker, Pluto. All of our paid features can now be purchased a-la-carte. Additionally, this PR supports granting communities an open-source/education/non-profit status which grants them a free private channel and moderator seat.

[brianlovin]

Hey @mxstbr - in the midst of working through the payments v2 upgrade work.

I'm building this with some of the following requirements:

• we need to to handle a-la-carte features, which means that we will be balancing many subscriptions, many quantities of those subscriptions
• we need to handle aggregated billing; e.g. a community might have 2 moderator seats, the indie plan, and want analytics as well - that's 3 subscriptions, but they should only get one monthly invoice
• count-based features (eg. total moderator seats if user is on Indie plan and has 2 additional moderator subscriptions === 3)
• it should be really easy for us to create, update, and delete subscriptions when a user has many active subscriptions

I'm not happy with how v1 (current) payments work is implemented, mainly because we're doing the following:

• storing a confusing subset of information in our database, between the recurringPayments and invoices table
• have confusing methods to determine if a user is currently pro or is on a specific community plan
• I don't have faith in the underlying data to be accurate; we are essentially maintaining 2 sources of truth at the moment: our data base versus Stripe's database.

This first commit is my brainstorm of how an API might work here; I've gone with a Stripe class that gets invoked with a customerId - that class instance then stores things like customer records, subscriptions, and payment sources in an instance state - that state is then used to calculate things like hasAnalytics or hasPrioritySupport

I've tried to architect it to require as few network requests to Stripe as possible; it should truly only be one request to get a customer, but I have fallbacks to make new network requests to stripe to get the latest information.

If you look at that commit I set up a dummy route where you can see demo usage, and you could visit localhost:3001/api/stripeMock to test results:

const { Stripe } = require('../../models/stripe');

const Customer = new Stripe({ customerId: 'cus_CD6DWvSmbuVxUi' });
const hasAnalytics = await Customer.hasAnalytics();

Overall I'm fairly happy with this approach, and in theory should be easier to write tests against.

Here are key benefits in my mind:

• always refers to the Stripe database as source of truth
• ergonomic - I love just writing Customer.hasAnalytics() and letting the method reconcile itself internally to get the latest data from Stripe

Here's how this would affect the rest of the system:

• A customerId string would need to be stored on our end - ideally on the User type
• I would like to defer all community settings modules to use this new API; for example, instead of querying our invoices table to populate the payment history component, we'd instead just do something like new Stripe({ customerId }).getCharges() in the resolver

Here's where I'm stuck:

• We shouldn't hit Stripe to resolve the isPro field on every user; is there a better way we could store that value without having to keep a bunch of stripe data in our db? One idea here would be to store a isPro field on the user record in the database itself, and we just use Stripe's webhook system to alert us when they upgrade/downgrade.

Here's where I'd love your thoughts:

• I'm a bit nervous I'm going to fuck something up with all the async class methods; I've typed the shit out of everything and so far so good, but do you have any thoughts on my design so far?
• Do you feel it's bad to hit Stripe every time a community owner views their community settings? I don't see a problem with this, but it would in theory be slower and require more external network requests to resolve components versus just hitting our db.

Anything else come to mind? You can also just tell me if this approach is a bad idea in general :)

[brianlovin]

Oh, and obviously this isn't done - just a proof of concept! I'd need to write a bunch more methods and do a whole bunch o testing.

[brianlovin]

44325e3 is an alternative implementation that solves a couple problems:

• users don't have to remember to instantiate a StripeCustomer with new
• StripeCustomer is just an object that exposes a function to return a class; in theory this is extendable by adding new functions here

Example:

const hasAnalytics = await StripeCustomer.init({ customerId: 'cus_CD6DWvSmbuVxUi' })
    .hasAnalytics()
    .catch(err => console.log('Error', err.message));

console.log(hasAnalytics) // true

// note you can init with no arguments as well - not sure if this is confusing or not
const newCustomer = await StripeCustomer.init()
    .createCustomer({ email: 'briandlovin+test2@gmail.com' })
    .catch(err => console.log('Error', err.message));

  console.log('newCustomer', newCustomer); // customer object

At this point I'm mostly just doing research around how this kind of composition should work; I'm finding a lot of articles that would bemoan my use of class here, but I'm struggling to think of alternative ways to handle a stateful set of methods :/

[brianlovin]

LOL.

Just got off a call with Kelly.

tl;dr: I'm glad I wrote this PR because I learned a lot about JavaScript, but Kelly has like a 10x more sane and manageable approach to this that makes a lot more sense.

Here's how it's going to work (still open to suggestions, but this feels pretty sane to me):

1. Stripe is the source of truth and is constantly sending that source of truth to us via webhooks
2. We always want to read off the source of truth, but never want to rely on Stripe's uptime or suffer the consequences of network request latency
3. As a consequence of this, what we want is our own internal store of the source of truth which is pushed to by Stripe
4. Given this requirement, we will want to create new tables in our database:

• SpectrumCustomer
• SpectrumCharge
• SpectrumInvoice
• SpectrumSubscription
• SpectrumCard
• Spectrum[InsertStripeProperNounHere]

5. We set up a new worker (yay!) whose only function is to listen to inbound webhook payloads from Stripe. Based on the payload event type, we transform the data into a near-identical version that gets stored in our database.
6. As a result of this, we can basically repurpose the entire class I wrote above, but swap out the spot where it requests data from Stripe with a database request against the appropriate table in our DB! Yay, I didn't waste all my time!

Still totally open to feedback on this @mxstbr but that method feels way more sane to me: we can isolate each webhook event into its own function that mirrors the source of truth in our db, which means we get to keep all of the same logic we have in our resolvers for checking user/community features! The cost is more storage in our db, but the pros are huge to this approach by decoupling our resolvers entirely from Stripe's infra.

[brianlovin]

c8b2db7 is a proof of concept of the above messages.

Ploutos, "Wealth," appears in the Theogony as the child of Demeter and Iasion: "fine Plutus, who goes upon the whole earth and the broad back of the sea, and whoever meets him and comes into his hands, that man he makes rich, and he bestows much wealth upon him."

So this worker is called Pluto, the purpose of which is to make us dat money.

It works by receiving Stripe webhook events, figures out a handler, and then resolves itself with a 200 response to Stripe.

If this looks good, tomorrow I'll set up the tables proposed above, the transformation layer (eg stripe customer => SpectrumCustomer), and start implementing handlers! They should all be fairly straightforward, and I'll only work on ones with the highest impact; we can do cool stuff here down the road about dispersing async jobs to hermes to send emails about failed payments, expiring cards, receipts, etc (and remove the /api/stripe endpoint from iris!)

[mxstbr]

Haha damn, you went well in the weeds there! Glad you learned a ton about JavaScript 😊 The worker implementation makes sense to me. 👍

[brianlovin]

Okay, I'm going a bit in circles here, mostly because we keep coming up with complex edge cases that break some of my original thinking.

Here's a new approach we could explore:

1. When a user creates a community, we auto create a customer in Stripe that assigns a customerId back to their user record in our db
2. In order for that community to create private channels, add moderators, view analytics, etc. the community will need to have a card on file. In order to do this, we will create a 'billing' tab where a user can store payment methods, update them, delete them, and have a default.
3. On the first of each month we run a cron that loops through all of our communities and figures out what features are being used; as a result we can do the following:

• auto-move people to the appropriate tier based on feature usage. e.g. we see that they have 1 private channel and 1 moderator, they should be billed at the indie plan level. But if they have 2 private channels and 1 moderator they should be billed at the indie plan + an additional private channel

4. That cron generates a Stripe charge, which gets paid via the card on file with their community.

The outcome of this approach is the following:

• We never have to worry about plans, subscriptions, aligning billing dates, proration, etc. All we care about is: does this community have a card on file?
• We can automatically apply "tier" discounts by just counting the features being used and figuring out the most cost-effective way to bill that user
• We never care about billing events while a user adds/removes features. I.e. if they go on a spree and add 3 moderators today, then remove 2 of them tomorrow, then add 100 the next day, then remove 99 the next day, we don't have to process anything - we only care about the state of the account at the time of charge generation
• Because a user can have multiple communities, and multiple communities can have multiple different combinations of features, we could end up having a nice combined bill at the end of the month; I think I could make it so that the same customer could user a different payment method for different communities; although this is pretty edge case, we already have an example: us!

Some downsides of this:

• People can figure this out and just remove moderators/freeze private channels before a bill event, then re-upgrade after a bill event. I'm okay with this being a "risk," because it would be easy to detect this behavior down the road when it's at a volume we care about

Some data model changes we'd want to consider:

• create a customer whenever a community is created, store customerId on user
• we'd need a new way to track if the community opts in to analytics or priority support, maybe just a boolean on the community record?

What do you think?

[brianlovin]

The more I think about this, the more appealing it is in terms of managing complexity at this early stage. Basically we can just stop caring about counting moderator seats, checking to see if a private channel is available to post to based on the community's payments records, etc. Instead all of the features just work without needed all this extra permission logic...then we just bill based on what's being used each month.

[brianlovin]

Update: in touch with a field engineer at Stripe who has some ideas he wants to test on the use case I've attempted to implement, and the one describe above. Will follow up and work on something else in the meantime.

[brianlovin]

Quick note on progress here @mxstbr for feedback:

1. I wrote a migration script to go through our communities and set an administratorId and administratorEmail field based on the earliest owner record. Loved how easy rethink made this. I know you were hesitant about these fields, but I feel it's important for us to retain some record of who actually owns this community, and the benefit of keeping an administratorEmail field is that community owners can change this so that personal notifications go to a different place than community notifications (i.e. billing events)
2. Pluto now has 4 changefeeds that work like this:

• when a community is created, we create a customer in Stripe - we do this because it will just make our lives a thousand times easier when handling billing logic and queries, if we always assume that the community is matched to a stripe customer
• when a community is edited, we update the metadata for that Stripe customer (super small detail)
• when a community is deleted, we close the Stripe customer and automatically cancel all subscriptions (again, this will just save us some headaches of potentially forgetting this down the road)
• when a community administrator email is added or changed, we update or create the customer in Stripe to have the right email. We need this because users will likely want billing emails to go to a different email than their personal email account

As a result of these changes, and after the migration, we'll have the following data scenario:
a good handful of communities on Spectrum will not have customer accounts on Stripe because the owner does not have an email on Spectrum. I will have to add a check against this in the UI, as well as support a mutation to add/update the administratorEmail field - not that our changefeed in Pluto will auto-detect when an administratorEmail is added and auto-create the Stripe account.

Between stripe webooks and this changefeed, we will always be in sync with Stripe.

[brianlovin]

if we always assume that the community is matched to a stripe customer

Actually I need to change some code so that this will be true, since I was originally trying to ensure that all stripe customers have an email on file; but this doesn't actually matter for us as long as we require an admin email before they can create any subscriptions.

[mxstbr]

Don't leave the .only there 😉 Danger will complain!

[brianlovin]

Whoops! Forgot to remove, thanks :)

[mxstbr]

Excited man. Let's get this on alpha!

[brianlovin]

Wow I have no clue wtf is going on with these tests, so many are failing but untouched in this branch.

Additionally, I'm totally unable to run any tests locally:

Curious @mxstbr if you're able to run tests locally on this branch?

[brianlovin]

A bit annoying that everything passes locally but fails in ci, I think because of a db mismatch :/ Commenting out the problematic test (that passes locally) for now to just get this to pass, and I'll re-integrate it into my other tests branch later.

[brianlovin]

Yay! All tests pass <3