by arnon on 2/26/24, 11:41 AM with 213 comments
by jchw on 2/26/24, 12:28 PM
However, I will admit, I had a hearty chuckle at this line:
> "why can’t we just dump a file of what we need to bill on S3, and have a CRON job pick it up and collect payment?"
Under no circumstances does my engineer brain think this is a good idea. At all.
But, I will dump one aspect of my engineer-brain thoughts: My favorite "billing architecture" decision is to try to decouple billing as much as possible from credit in a system. For example, if you have a subscription system where the user pays ahead for a given billing period, I prefer to have the entitlement itself just store the expiration date and the details about what entitlements the subscription grants during the time period it is active. The billing system can store the subscription and sync back to the entitlement as-needed. This makes both manual billing by human operators (not to mention debugging and patching around momentary issues) and something like a Stripe integration very easy. You should, of course, be very careful to leave it open for extension in the future, but this seems to be a very nice decision that doesn't, in itself, limit you too much.
Obviously, this is not my original idea, but it's still something I've grown to like a lot, especially after having tried other things less successfully.
by chasd00 on 2/26/24, 12:43 PM
by ivanmontillam on 2/26/24, 6:28 PM
Billing systems are of high complexity; I recognise that. However, if Chargebee, Solvimon, Stripe, Recurly, Orb, Metronome, Lago, Togai or anyone else has that body of knowledge, we could instead collect that knowledge in one place.
Indeed, there's no better approach than the one that serves you. If you're a subscription-based SaaS, you have specific solutions for your business. If you're a usage-based API, you have specific solutions to the billing.
But we could have all that knowledge, approaches, paradigms, programming patterns, better and best practices in one place, instead of discouraging the practice. There are also edge cases where a company is not U.S.-based or European, and a billing solution like Stripe wouldn't work, e.g. your company is based in Venezuela, and you can't have a Stripe account. What do you do in that case? You must forcefully build your own billing solution and connect it to the local payment gateways with their arcane SOAP-XML APIs.
--
On a separate note, "building your own billing system" reminds me of the topic of "rolling your own SIEM" with the typical Elastic + Grafana setup.
I don't recommend it, but I understand why it's such a hot path for an IT Security department to do it.
by nickjj on 2/26/24, 1:17 PM
You need to track affiliate codes back to sales and a user, handle sending payouts to an affiliate at their configured payment provider and if you want to go all out then handle metrics around visitors and present a UI to the affiliate so they can see their conversion rates and payout history.
Fortunately most of that can be built incrementally. As long as you associate a unique code to a user and wire up associating that to a sale with a specific commission % amount everything else can be done manually or skipped. For example I pay affiliates out once per month where I goto Zelle or PayPal and send out the payments. It takes less than 10 minutes. There is no front end for tracking conversions and it's also never been a cause of someone saying they don't want to be an affiliate because of that.
by 6510 on 2/26/24, 1:05 PM
You really want to have "the order" as static as possible, for "the invoice" there is even legal obligation.
Then it goes something like...
I bought 5 and got a discount but I want only 3 and ohh there is a typo in my name and one in my address - sorry about that.
No problem, you credit the invoice, make a new one for the same amount with the name and address correction, then you wait for the items to be returned and checked and credit it again, make a new invoice again. The return shipment stays in limbo for a while, they order 5 more, one arrives broken, they want a refund rather than a replacement. With 2 delivery dates the system doesn't apply the 5+ discount.... and all of a sudden you have tons of transactions and "paperwork" for what you initially imagined to be 2 simple orders. They also forgot their password so they made a new account using a different email address. Looking over the logs a year later it is hard to figure out why they got a discount for just 3 units.
by mattbee on 2/26/24, 12:53 PM
You do need to understand the concepts of invoices, credit, tax periods, pro-rata billing changes and so on... but all of that knowledge can be used to make an informed decision on build vs buy, rather than an automatic reason to outsource.
The only external API you need for software-as-a-service is a credit card processor, two if you're fancy. Sure after the first year you will probably have a bunch of manual work to do and your accountants will tell you the dumb stuff you've done, and you'll learn a lot about accounting :)
(I would still shop around to start a new business today, but with the confidence that building isn't very scary)
by kyrra on 2/26/24, 6:43 PM
1) Month/Quarter close. While this post talks about the account ledger, when you start dealing with a public company that has to report numbers, you have hard-cutoffs and have to make sure everything goes smoothly for month or quarter close.
2) Cash-in-transit accounting: this post assumes your payment processor is 100% correct and nothing goes wrong. When you get big enough, you need to be able to match any money that lands in a bank account with a given invoice/billing entry. You need to be able to detect any invoices that do not have a paired bank statement line. And as others have called out, the reverse can happen where you get paid for something that may not have an invoice item associated with it. Being able to deal with credits on a bank account not associated with a billing invoice can be equally important.
Maybe you can say that these are all accounting departments problem, but they are tightly coupled and the 2 teams need to work together that there are no discrepancies between their books.
by jdwyah on 2/26/24, 2:56 PM
I was just coming to hn this morning because I wrote about using FF for entitlements: https://prefab.cloud/blog/modeling-product-entitlements-with... I took some inspiration from another of Arnon's posts about SKU format for the post.
FF don't seem like the perfect place for entitlements, but in my experience they're often the best tool at hand to deal with the challenges. I'd love to hear alternative opinions.
by d_burfoot on 2/26/24, 2:57 PM
Perhaps the general case of X is enormously tricky and complex, but in my use case I only need to handle a specific subset of the complexity. Therefore I can build my own solution that only handles the complexity I need, and it will be much simpler than off-the-shelf tools.
I absolutely adopt this stance for X=datetime. My approach to datetime requires two function calls to be provided by the library: convert an epoch time to an ISO formatted time string in a particular TZ, and the inverse. I never touch any other library code; I do all other time manipulation in my own code, in terms of those two functions.
by talkingtab on 2/26/24, 1:42 PM
Many people think it is a good idea to close your eyes and just start walking, resulting in untimely injuries or even death.
1. Countries and even cities have different rules for crossing the street. 2. You may not know this yet but you need to look both ways. 3. Many people do not realize they are color blind, resulting in death and injuries because they cannot tell red from green. 4. You may look both ways, but do you look up as well? In cities people can drop things from windows, in the country birds may fly over head. More and more space junk is falling to earth. 5. Thing may come at you from multiple directions. 6. etc.
Not saying this article is that bad, but I would appreciate an article framed as a how to. I am much more likely to read articles that are not framed negatively.
by tombert on 2/26/24, 6:42 PM
Most of my job involved working on their marketing pages, but at some point they had me work on the credit card processing platform, and I've sort drawn a soft line in the sand that I won't do that ever again.
Part of it was just that the code was really messy (it was ColdFusion after all), but a lot of it boiled down to having to deal with the million edge case conditions required to achieve PCI Compliance. Somehow, that company had managed to get a PCI Compliance label, and I have no idea how, because the code was held together with duct tape and prayers; there were hundreds of nested if statements, and if's nested inside else nested inside other ifs.
I'm not saying that it's unnecessarily complicated, but I know I'm not smart enough to deal with it.
[1] I'd like to point out, this was 2012; it was considered kind of outdated even at the time!
by spintin on 2/26/24, 2:12 PM
itch.io is a good alternative if you sell digital assets and even services as you can verify payments with mail.
by msluyter on 2/26/24, 3:04 PM
* Forward billing vs billing in arrears. We had to treat different customers differently.
* Tiered billing -- e.g., if a customer used > X amount of services, they got a discount.
* Special rules around when billing starts for new customers (e.g., no bill for the first month).
There are probably more I've forgotten. We billed by data usage, mostly, and I used to say that their bill was the integral of the customer's usage graph over a month -- which was true -- but that explanation didn't gain much traction with the accounting folks.by bearjaws on 2/26/24, 1:05 PM
Absolute nightmare when we blew up, had to hire 6 people to beef up the billing side of the website and it definitely cost us way more than those 6 people due to mistakes.
I ended up working on parts of it for our customer service portal, lots of band-aides applied in a rush as we scaled.
by Aspos on 2/26/24, 4:15 PM
Billing problems in general are a subset of problems one has in banking because banking system basically = billing + interest calculation + payments + accounting + multiple equally complex things and it has to work in real-time in a super-regulated environment while working in perfect sync with other systems you don't control.
If you do everything just right and manage to keep the can rolling down the street for sometime, there will be a moment when regulation changes or your business changes and your architects suddenly resign and you will end up with a mess which requires more engineering dollars than off-the-shelf system would.
Don't build your business on such foundation. It is XXI century already, no need to re-invent stuff which is readily available for peanuts.
Yet, every new neobank comes up with their own shiny corebanking ledger because they can't be bothered to look into somebody else's petrified spaghetti.
by llamaLord on 2/27/24, 6:33 AM
The biggest mistake I see companies making is not properly appreciating the fact that there is in fact 4-5 entirely independent business processes going on inside that monolithic concept they refer to as a "billing system", and they REALLY need to be kept seperate.
IMO the fundamental seperation that is absolutely critical to make is distinguishing between your "entitlements system" that tracks what SKUs you offer and which customers have what SKUs (and in what quantity), the "accounting system" that takes input from the entitlements system and tracks the over-time net balanced owed by the customer, and finally the actual "billing system" which periodically takes the balance from the the Accounting system, zeroes the account back to nil, and generates an actual point in time "bill" for the customer to pay.
So many systems don't properly respect these boundaries and suffer the unbelievably painful consequences.
by Vinaydatta2020 on 3/4/24, 3:38 PM
by sbrossie on 2/26/24, 11:49 PM
by gmfawcett on 2/27/24, 2:25 AM
> Sometimes I see other people fuck up a project over and over, and I say “I could do that better”, and then I get a chance to try, and I discover it was a lot harder than I thought, I realize that those people who tried before are not as stupid as as I believed. That did not happen this time. Moonpig is a really good billing system. It is not that hard to get right. Those other guys really were as stupid as I thought they were.
by wolfspider on 2/26/24, 5:20 PM
by godzillabrennus on 2/26/24, 2:27 PM
Very unique circumstances but huge undertaking.
by etewiah on 2/26/24, 1:20 PM
by atonse on 2/26/24, 2:13 PM
But in all seriousness, we looked at Stripe, thought about "aw man is it worth spending the .5% extra" and then spent about 30 seconds thinking about how much time it would take to roll our own, and it was an absolute no brainer.
(replace Stripe with any of the other similar competitors, not trying to be too biased, but stripe is kind of the mind-share default for SaaS startups, aren't they?)
by whydoineedthis on 2/26/24, 1:12 PM
by WirelessGigabit on 2/26/24, 4:35 PM
Sounds like an engineer with not enough experience. We all did thought like this once.
This is a solved problem. Don't roll your own queue / job system where you inevitably run into transactional problems. Use something like Airflow.
And mentioning S3 in this sentence has no value. S3 is merely a way to store the data temporarily. But using S3 immediately makes us not consider (more suitable) alternatives.
by welder on 2/26/24, 8:46 PM
by 6510 on 2/26/24, 3:47 PM
by wdb on 2/26/24, 3:13 PM
Sometimes you had to reimplement whole systems that SAP comes out of the box with. Thanks not to be named fashion brand.
by stevev on 2/26/24, 12:59 PM
by epberry on 2/26/24, 2:30 PM
by _pdp_ on 2/26/24, 8:09 PM
by paulhart on 2/26/24, 3:39 PM
Really?
Odds are that it's not, and therefore you should farm out that work to someone who _does_ make it their job. There are reasons why companies implement software from Microsoft, Oracle, and SAP, including that it's better to reduce costs on things that don't differentiate you in your market (and it's nice to have someone to "pointedly talk at" when things go wrong).
by estebarb on 2/27/24, 3:42 PM
by bombi on 2/27/24, 3:09 AM
by Havoc on 2/26/24, 11:14 PM
by zengid on 2/26/24, 5:47 PM
by coolThingsFirst on 2/26/24, 6:07 PM
by zie on 2/26/24, 3:56 PM
by rahimnathwani on 2/26/24, 10:42 PM
With that approach, how do you:
1. Make your ERP recognize revenue correctly? An ERP system isn't magically going to know how your revenue should be amortized based on certain events, or what should happen when a partially-amortized order is later upgraded/downgraded to a different package/plan.
2. Ensure that your refund/upgrade/downgrade calculations are consistent between product systems and the ERP system.
Even if you emit all the necessary events (and each contains all the necessary data), there has to be some system that's programmed (or, at least, configured) to apply your revenue recognition rules to the event stream. Given how much difficulty people have even describing how things should work, I don't think you can just abdicate the revenue recognition to another system and just assume whoever configures/programs that system will do it right. Even if you can assume that, it's equivalent to saying "it's complicated so you shouldn't do it yourself; have someone else do it".
by jongjong on 2/26/24, 11:05 PM
The amount of money I've seen startups forking out annually for various billing and platform subscriptions would have been enough for me to build them a better system from scratch in under 6 months. The amount of money that companies waste on rents is ridiculous and it's sad because that money could have gone to skilled engineers instead of hotshot entrepreneurs.
For my own SaaS startup, I built the billing system from scratch, it measures every millisecond of CPU time and every operation used by each account and adds them to the current active bill. At the end of the month (or whenever), I manually trigger a process to close off active bills and the system will automatically display that one as due in the UI and will open up a new one and new usage will be recorded against that new active one.
The trick is to use a function which automatically figures out what bill to use based on a specific flag so you don't need to worry about the implementation details of which bill to record usage against. This function should handle all possible situations; including the few milliseconds of delay which can exist between the old bill being closed and the new bill being opened.
That's where idempotence can be useful as mentioned in the article; that way if the system fails to record usage against a new bill because (for example) that bill was already just created concurrently by a different process then it will retry again later and record the new usage stats against that existing bill.
The trick I use is to have deterministic IDs for bills that are based on the nearest whole unit of time in UTC (rounded up or down); the amount of rounding I use allows me to control the allowable delay between concurrent processes and determines the amount of pending usage data which is kept in-memory within each process before it is flushed to its bill in the database. If the chosen unit of time is a 'whole minute' then it means that it's not possible to generate two bills within 30 seconds of each other by concurrent processes. This is greater than my database update timeout so it effectively guarantees that two processes cannot accidentally create two bills for the same period. If a specific process took longer than 30 seconds to update their usage info, that process would reconnect and either realize that an active bill has now already been created by a different process or, if not, it would try to create it again with a new ID corresponding to the new 'whole minute'.
I have full flexibility to change my billing to any time interval I want. It doesn't even have to be the same for all users.
Anyway, as you can see, it's very simple.
by zackmorris on 2/26/24, 6:21 PM
I found the documentation both overly verbose and lacking, so it took me longer than it should have to understand basics like subscription item ids and the 50-100 id types which people have collected into lists. Then there are simple bugs like how the go live button only shows in test mode, with no simple way to copy live mode data back into test environment(s), so developers have to manually export/import stuff that business people set up in their attempt to save time. The most basic features are missing, like a way to copy any object as json from the browser dashboard, forcing us to look down in the logs/events to copy the latest version, often with wildly inconsistent/unpredictable formats.
And Stripe does little to support actual real-world use-cases, for example: subscription events come through webhooks ok, but subscriptions have a state like "active" or "incomplete" (meaning that payment hasn't gone through yet), causing subscriptions to become stateful. Meaning that instead of a user being subscribed (yes/no), the app has to consider additional criteria and edge cases around missing or lapsed payments. And there doesn't appear to be a synchronization mechanism for eventualities like a backend server being down for a day, causing events to be missed. Stripe does a best-effort resend some low number of times, but then the events are lost. It's up to the developer to diff the backend's database with the Stripe subscription list and then sync individual subscriptions manually.
These are all just exactly the kinds of bugs/features I predicted would be there before I used it, which is why they felt like such a slap in the face. I suspect that there are conceptual shortcomings throughout nearly every service provided by Stripe, and I'd be over 50% confident that any ones I predict here would be found upon first use. These are artifacts of favoring agile over waterfall for formal engineering challenges.
Basically what I'm saying is that I consider the Stripe API to be what an MVP payment processing system might have looked like back in the 1990s under older paradigms like SOAP/XML. But nobody has created a wrapper for Stripe yet that works how people expect, more like Venmo/PayPal that maps customer use-cases to CRUD operations.
I thought that maybe Laravel Cashier would do some of the heavy lifting, but it appears to be just a thin wrapper over Stripe, with its own set of oversights.
I know that everyone uses Stripe and apparently loves it, and I applaud its efforts and accomplishments around reducing the friction of payment processing. But I can't help but feel that there is an element of having to drink the kool-aid here. I would still recommend Stripe though, so this rant is directed more at its developers who may not be aware of these issues. Stripe would do well to perform some new user testing like Apple used to do when designing its human interface guidelines, to enumerate the pitfall(s) in each step of Stripe onboarding.
by idlewords on 2/26/24, 2:50 PM
by huqedato on 2/26/24, 2:13 PM