I was working on a billing system for utility company and it was a nightmare but not for the reasons from the article.
Dates are no problem, everything is billed at the end of month, quarter or year.
Upgrades are downgrades are simplified, customer can legally change tariff only on predefined dates/events.
Usage - yes, but exampled in the article do not even scratch the complexity of this
Idempotency is not really needed, the billing process is not continuous, it is a batch job.
Cash collection is out of scope of billing software.
Taxing was clear. Yes, there were multiple billable items with different tax rates, but not too complex. All customers were local.
The nightmare part was:
- customers can have multiple consumption locations, location can have multiple meters and customer can request to split the bill on multiple invoices or combine multiple bills to single invoice as they please
- meters can be replaced at any time
- meter readings are available on dates totally independent from billing cycle. Most of consumption data are mere forecasts.
- when the actual consumption data is available, everything must be recalculated AND compensated (on a correction invoice showing the difference)
- actual consumption data can be wrong and may be corrected at a later date, even multiple times
- consumption points can be added or removed or moved to different customer at any time, but this information is only available after the fact
- the prices can change mid-billing cycle for some customers but the consumption data is not available with that granularity
- customer legal information (company name, address) can change mid-cycle
Indeed, we only scratched the surface of the complexity of metered billing, we'll do a deep dive soon on this topic (it does deserve its own post).
I think for the'nightmares' you mention:
- some might be specific to utility (not applicable to an online business), such as '- meter readings are available on dates totally independent from billing cycle'.
- Some topics might be simpler for a utility co: tax for instance (you know where your users are, and they are limited to a geographic region)
- But some nightmares such as '- the prices can change mid-billing cycle for some customers but the consumption data is not available with that granularity' do really resonate for online businesses too, indeed
Thanks for sharing, great insights for our future post!
For online businesses as well, selling above a certain amount of revenue or product can trigger what's known as a nexus event in taxation wherein the business becomes liable for collecting sales or use tax. Now online businesses are on the hook for collecting sales tax in every jurisdiction above wildly varying threshold values. Should the business fail to collect the sales tax for selling from Montana to California for instance, then the business will be liable for sales tax not collected for products or services sold to California residents online even though the business might physically reside in Montana.
Is there an easy way to deal with this e.g. compliance as a service?
Sounds like a nightmare to deal with, of course this is only assuming it is enforced. If it is not well who cares, companies like Lyft/Uber view laws as a starting point in negotiations when expanding into new areas not as a brick wall of dogma.
> Is there an easy way to deal with this e.g. compliance as a service?
I think we call ourselves CPAs :). In all seriousness the 'solution' depends on what industry we're talking about. Short term rentals? Avalara. Software as a service? Various accounting firms and startups. Etc.
Did the company use a relational database to deal with the billing and subsequent adjustments? I've been there: it can be more straightforward to use an event streaming approach that recalculates billing as new events arrive.
Yes, almost everything ended as temporal or bitemporal model. Pain to work with. Even for simple questions like "What is the customer #1234 company name?" the system needs to know when are you asking.
In field metering situations you can have the timeline of each meter, when it records data points, and then you have the different timeline of the backend service, when it actually _receives_ data points.
Bonus points if the meters can send data out of order.
I did a tender for aggregating events and emitting notifications from a fleet of transport vehicles that reported metrics over LTE.
When connected they would emit events as they happened, but when there was poor service or disconnection they would buffer them and send them as a burst once they reconnected... in reverse order.
I forget the finer details, but the service I was writing was supposed to notify of trucks entering and leaving geofences, inactivity etc. anything you can think of where the order of events is important.
In a time-series (or event) table you have one date - when the data point is valid (event occurred).
In a temporal table you have two dates - interval when the data is valid.
In a bitemporal table you have two intervals. First describes when the data is valid in real world, the second when does the system know about the real-world state.
I.e. if company changed address on 2022-05-01 but the system will not get that information until three day later (2022-05-03) you have two validity intervals.
Then you may ask - what was the company address on 2022-05-02? And what did the system think the address was on 2022-05-02? You will get two different answers. This is important if you need e.g. to make a correction to previously issued invoice. You need to know what data was used for original invoice and what data should have been used.
In my opinion, the best way to do it is ingesting events in a relational db with an idempotency key to prevent duplicated events. Some companies don't need real time billing data, so you could just use it to aggregate these events with the aggregation function that fits your need for a certain feature. Thus, you have the flexibility of (i) calculating pretty much everything you want, (ii) give granularity of usage to your customers and (iii) be closer to real time (you play the query anytime you need it).
I was also consulting for a utility company. Billing was easy, we used SAP (lol). But CRM (where we also used SAP) was a nightmare. The different processes where a nightmare. I mean utilities are amazingly complex logistically. New metering for a single house build, when it's ordered, installed, etc.
I work in a closely related space: parent's comments are accurate. Some of these items can be solved with better forecasting models, but the issue is that almost no-one actually cares about 'better' forecasting at inidividual meter levels. Sysytem-wide (or at least substation level) forecasts are well studied for supply/demand considerations (at a minimum). Also, the variability in consumption patterns at an individual level are large. IOW, one might be able to generate forecasts accurately for 'typical' residential consumption (at a per-household level), but commercials can be very different.
Ideally, IF (and this is a BIG if), forecasts for "most" premises was accurate enough, then one could make the claim that the missing data shouldnt really cause a recalculation of bill once the actual consumption data comes thru, except in cases where the error is large. Guess that means that the error check/correct process needs to continue to exist.
The meter being replaced shouldnt be a major issue, this is relational (at least we treat this all as relational) data that should be captured (by the customer information 'CIS' system), and should be available with a 1-2 day delay. Similar argument applies to other relational aspects of the premise under consideration. Not saying that those are easy (more ways for this process to have gaps).
I've worked at two companies in their payment systems. This post does a good job of mentioning many of the billing system challenges. I think it understates how difficult and important idempotency is to maintain at scale with multiple teams though.
Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)?
He also did not mention fraud, which is basically uncollected spend that results in charge backs typically. If you get too many chargebacks, you'll get fined. If you get even more chargebacks, you'll get kicked off the card network and will no longer be able to accept cards within that network.
Post pay (customer gets the product and pays afterwards) models are subject to more fraud, uncollected charges, etc.
There's also the fun of dealing with downstream payment processors. I haven't worked with a processor or bank yet that was very reliable. Sometimes they'd return 500 http codes and then process a payment, requiring manual intervention. Sometimes that happens with files of payments because most banks deal with files for transactions.
> Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)?
IIRC, I've read this is one of the reasons that systems like SAP are so popular. They might be an ugly monster from a technical perspective and seem overly complex, but they provide thoroughly tested implementations that handle all that stuff.
Hey @tablespoon, I'm one of the cofounders of Lago here.
Products like SAP do seem complex because of (i) the complexity of the billing problem and all the edge cases, (ii) they were not built with an API-first or engineer-first mindset
We're only at the beginning of the journey, but that's one of the reasons why we thought of open-sourcing our billing API, so that the work could be forked and tweaked to address edge cases if needed (no need to wait for SAP to update the product: and they will never prioritize small edge cases).
Also, it seemed to be no middle ground between 'Stripe Billing' (it's great for B2C Subscriptions) and billing systems that could address the millions of nuances of pricing between 'subscription' and 'usage-based' (most companies are a mix of both).
In the SAP R/3 family of products, there is also a component Factory Calendar (IMG -> SAP NetWeaver -> General Settings -> Maintain Calendar, transaction /SCAL) - a component that knows about all the various calendar systems incl. public holidays of the world and the dreaded leap years. If I remember correctly, it's customizable for each customer premise with respect to whether each day of the year is salaried or not.
This is a component classified as "basis technology" relied upon by many modules for billing (e.g. should a daily salaray be calculated in a country for a particular person? Should an invoice be considered "overdue" yet, based on the number of business days passed since issue? etc.).
I am the last one to defend SAP, but they do have an API first approach since some time, check this out: https://api.sap.com/products/SAPS4HANACloud/apis/all Also edge cases are taken care of, sometimes before you even small updates, via notes, mostly tax regulation changes. So yeah, for these things SAP is really made for. If you have a SAP account, you can check out, just to see how many changes are coming in: https://launchpad.support.sap.com/#/solutions/notesv2/?q=pol...
It's not like SAP/Oracle/etc have some magical power. They just pay attention to these things, build the product, THEN release the product. And initial releases are buggy just like most software.
So more agile players can get involved more quickly and entrench position.
Hey @Brystefor, I'm @Rafsark's co-founder.
We could have spent more time on idempotency, indeed.
We have not deep dived into the 3 other aspects you mentioned indeed, we're actually planning to
1/ open-source the articles and add additional challenges as they are mentioned
2/ open our roadmap based on these inputs as well
Thanks for sharing! Will ping you back here when it's live
PS: we had not thought of financial regulation changes indeed!
Oh, I didn't even realize this was a blog for an open source project. I assumed it was someones personal blog.
Two other things that you may want to consider:
1. PCI compliance. If you're involved in the storage of payment information, you may want to look into these regulations.
2. Regarding point 3 of your mission "...only integrating with specific partners to lock you in (e.g., Stripe Billing only working with Stripe Payments)...". I think there are many companies out there who want to integrate with more than a single payment provider (eg Braintree, Stripe, Adyen, etc) but run into this problem. It increases the cost of payments due to the "rent-seeker" charge method they use. Providing a design framework for how a single company could integrate with multiple providers could be beneficial. This is something I've considered trying to build, but I think it could fit with this initiative.
Hi @Brystephor! I'm Anh-Tho, one of the cofounders of Lago.
We're building an open-source alternative to Stripe Billing indeed, but we want to share our personal experience, hence the personal blog post.
We'll open our repo very soon and our public roadmap too. Thanks for adding the 2 points, we're working on 'a design framework for how a single company could integrate with multiple providers could be beneficial.' indeed!
1. The purchase happens in Texas.
2. The payment cleared in a specific 2 day period.
3. The item cost < $100. The cost of shipping has to be included in the item's cost.
4. The item falls into specific categories that almost certainly don't match whatever categorization system you are attempting to use.
Seriously, calling out just one example, https://comptroller.texas.gov/taxes/publications/98-490/scho... points out that a kit containing school supplies qualifies as long as the cost of the qualifying school supply items in it exceeds the cost of the non-qualifying items. I guarantee that you don't have a category for that in your accounting system...
Our team was building a telco operator in the U.S. We already did that in several European markets and in Australia - so we assumed that we were mildly proficient in taxation issues. And then came the U.S. - a complete nightmare of overlapping rules that sometimes are not properly published. In essence you cannot do this on your own - you definitely need a tax service provider (like Avalara). Funny thing is though that you still are pretty safe to be noncompliant sometimes somewhere and the provider will not take on the liability if their data / computation was wrong. In short: a mind-blowing disaster. And you had to PRINT your tax forms and SNAIL-MAIL them to every jurisdiction that you were doing business in. I mean: hundreds or even more. Every. Single. Month. And don't get me started on the banking system stuck in Stone Age. I definitely would think twice to set up a business in the U.S.
I worked on a tax app. Total nightmare. I remember having to adjust tax rate for a couple parishes in Louisiana and a city in Florida for example. Trying to come up with a readable, testable, way to get that done was something I tell ya.
similar. worked on invoice app having to apply tax, and working with existing foxpro(?) tables that were in postgresql. Trying to reverse engineer without having access to original code, and make things work "like they used to". Someone manually pulled down some free tax thing once per month, but there were so many exceptions that people knew about that were just hardcoded in to the code that I had no access to that I could never make it "just work".
We transitioned to using an external tool (taxjar, IIRC, but might have been something else). Was a pain to feed them data to get back tax data, and.. so many exceptions and exemptions I had to learn about. Weeks later I get "all these numbers are wrong".
I dug in and went through logs.
"no, these are right. taxjar is adding x% on ABC and shipping".
"We don't charge tax on shipping".
"Well, taxjar is doing it, because it's how you're supposed to do it in this state".
"We've never charged tax on shipping"
"Well... you've probably been non-compliant for years then".
I think when they'd started 'computerizing' this back in the early 90s, shipping costs weren't taxed. Or... (IIRC) the amount of shipping cost taxed was proportional to the amount of taxable goods being shipped in the state (vs out of the state) and when they started, what they were shipping wasn't mostly taxable (like... some of the main items were taxable, but the supplies weren't, or something similar). And... if they were shipping to a non-taxable entity (non-profit) then the in-state shipping charge wasn't taxable, but they were supposed to pro-rate it based on the proportional distance in the target state, if that state charged tax on shipping, then track that and remit to that appropriate county/state authority.
These were some of the rules that a variety of people had tried to describe to me both what they thought was supposed to happen and what they thought the original software was doing, but... they could never find 2 examples to line up to confirm or justify those explanations.
Another example: certain items (chemicals) were taxable in some counties and not others, and might be taxed differently based on whether they were being 'used' by technicians vs delivered to a customer (again, different counties had different rules - and some of these rules were indeed confirmed by digging through taxjar and their support and state agency websites).
When this company started, it was one location in two counties in one state, and as they grew, the specifics grew (and also changed yearly) and what they should do became lost to everyone.
Moving things to an external hosted tax-as-a-service system was supposed to 'simplify' things, but it just introduced a lot of turmoil and headache, because routine numbers people were used to for years were suddenly replaced with "new" numbers which no one trusted.
We should demand governments to maintain and provide an API for their tax codes. Let them have skin in the game. If the taxes are incorrect, businesses can point out that it’s the government API’s fault.
Completely developed in the open, and with full test coverage.
This should be required to collect taxes. No API? No taxes. API down? Tax waived.
True. One is a more realistic reality though. And it results in this thing only being built once instead of a massive headache for all.
I have to imagine government workers are already writing software to identify if we’ve all implemented it correctly, when the could have just given us the code/api up front.
> I have to imagine government workers are already writing software to identify if we’ve all implemented it correctly
Because of the jurisdiction issue, even if each local (and for each location all relevant governmental tax authorities) had a clear API, interacting with them all, and dealing with the interactions between them recursively, is itself a large problem.
And from what I've seen from smaller jurisdictions, the automation software is all about simplifying a human-in-loop workflow rather than full machine-driven rules.
Since we're talking about a hypothetical solution, in theory this would have a spec and potentially even be further centralized.
I could imagine a world where the IRS ran the tax software for every taxing entity in the US. Each jurisdiction would have to configure their specs or potentially even codify the business logic. International would likely be tricky but it's similar to what is being done with ACH and other banking systems that are slowing moving to international. So, it's not impossible there's just not much of a push for it.
You might be interested in [Catala](https://catala-lang.org/en/), a domain specific language for implementing legislative texts. Iirc, they have worked on the french tax code, and are starting on the us one now.
Gee, that's bold, I feel like "try writing in plain English" is already asking a lot of the legal profession (fortunately some of them are taking care of asking it of themselves). I wouldn't want to push it!
Here is an irony. If most contracts were rewritten in plain Spanish, they would be understood by a higher portion of the general US public than the current legalese!
If you believe the statistics on literacy and multi-lingualism, it's a tossup as to whether Chinese or legalese is more widely understood.
True, but if the lawmakers writing the tax code bear more of the pain of the complexity of the tax-code (i.e. get complaints and warnings of their own devs that have to implement that API) then they might write better tax codes.
That is the point of the tax code, to be able to use it for political wielding of power e.g. the Chicken tax because LBJ was pissed at Europe that caused a 25% tariff on light trucks imported to the U.S. based on a retaliation for European tariffs on American chicken..
Every tax system is. Because the experts who help to develop it are the same people who get paid to help you file your taxes. Making it easier would lose them income.
It is a combinatorial explosion problem. Each taxing authority probably isn't too different from the norm, but in the US you are more likely to encounter many different taxing authorities that overlap.
Come to Brazil, we have taxes on products 'circulating', moving your products between logistic centers is OK, but if the product is lost or stolen, taxes are due (as if it was a sale).
Some tax regulations were considered illegal by higher authorities, but that won't stop a tax auditor in a local level charging them and potentially closing down your business.
Why would you do that when you could just lobby the government to make the tax code even more complex, then sell a service to handle that complexity for you under the threat of jail time if you get it wrong?
You're being facetious but I don't think this is a radical idea. Government already publishes the tax code in English language. This is just formalization of the tax code in computer code: More precise, testable and provable.
What makes you think that this is being facetious?
https://www.propublica.org/article/inside-turbotax-20-year-f... has a good overview of how companies that provide software for calculating taxes guarantee a market for their products by keeping the process of calculating taxes more complicated than it needs to be.
In Bangalore, India, workers put on Smith face masks as they posed for selfies with the man himself. Fittingly, the tour culminated in San Diego, the home of TurboTax, the software that transformed the company’s fortunes. There, Smith arrived at his party in a DeLorean, and as he walked a red carpet, cheering employees waved “Brad is Rad” signs. To Smith’s delight, his favorite rock star, Gene Simmons of Kiss, emerged.
I have no words to describe what I just read. That's straight out of the TV show Silicon Valley.
This actually sounds like a good use for smart contracts if we switch to a CBDC. Basically you pass your transaction through whatever state’s contract and it will automatically do the tax withholding for you.
I used to work at a company that managed the systems for back of house in fast food chains. We moved customers from excel to an online system. Omg when it came to doing labouring in the US. It’s scary.
If a person clocks in at 9:52M. The company only has to pay them from 10am. But in other states they must pay from 9:45am. There’s so much complexity in the us due to all states being different.
Even when the laws are comprehensible and you're not running at scale, existing accounting systems make supporting simple new requirements difficult.
Imagine a US LLC setting up their chart of accounts to track IRS deductible expenses. Coding expenses into this 17-account schema in the normal course of book keeping means taxes can be filled by copying the account totals into the corresponding IRS form field.
But later, the business wins an federal contract which reimburses certain "R&D" expenses and possibly some "overhead" expenses. Now not only must every expense must be categorized with one of 17 expense categories for the IRS, but it must also be categorized with one of 3 categories for our federal contract purposes. But wait, a small business loan application may require profit and loss broken down by a different set of categories. New and local tax authorities may impose other categories.
One way to address this complexity explosion is with multiple instances of the same accounting software; each imports the same bank account statement, then justifies it against a chart of accounts. But many businesses will break out Excel and hope there aren't too many expensive errors.
If new regulation is inevitable, why does our software pretend otherwise?
Wait until you deal with Brazil. Billings systems are not hard they take engineers that are thoughtful and long-term thinkers which is not conducive to today's software developer hires. Which is why posts like this are made, Its a market ripe for dominance.
Implementing and maintaining billing is my daily job for the last 6 years.
I would also mention topics like:
- Reporting (various data aggregation and audit reports)
- High reliability, nothing hurts company more than unability to bill their customers
- Rounding, this can be seen as a subset of "taxes" but it's much more complex
- Locating user, also can be seen as a subset of "taxes" but user can have country A set in profile configurations, country B on credit card and country C geolocated from IP address
- Timezones, issue everywhere but when we talk accounting it's super important
- Talking to moronic payment gateway providers. This is the biggest ones. I would love to just flip Apple and Google like Linus flipped of nVidia. Proprietary, poorly documented, "find out yourself by weeks of experimenting" bullshit with no easy way of getting technical support even when you bring those companies millions per year. Things don't work, deprecate monthly or weekly and expect you to be always ready to make changes. Some implementation make zero sense at all with complete paradigm shift of handling payments like between Google in-app pay and Google subscriptions.
And it's not "billing", it's Compliance + Bill Presentment (marketing!) where we are always trying to find the most profitable local maximum of explainability to Sales, Customers, and Management/Enforcement while also having hard checks to prevent or at least mitigate losses.
Wow, thanks for sharing!
We found out 'billing engineers' are very rare and really demanded, as it requires to be very detail oriented, technical, and business processes oriented.
But even if there is high demand (and I guess: corresponding high pay, as you mention), very few engineers are up to the challenge!
Once upon a time I was the first product person at a now-decacorn, and my first task was fixing the billing system. It was quite the monster, and we ended up implementing a combination of Zuora and an internal system, as there were some parts of the billing model Zuora couldn't handle.
I came away from this with one big lesson - if you're considering a complex billing model, consider the engineering implications first. With most products, engineering feedback gets taken into account - often product proposes something, engineering breaks it down, product realizes that feature x is vastly more complicated than they thought and not worth the effort, and the requirements are changed to simplify or remove it.
The one thing that never seems to be true of is pricing models - that decision gets made at the very top and handed down with no chance for feedback. I think that if it the folks designing the billing system realized the costs, they might simplify things. If the complexity of your billing system means that 3% of your engineering team (plus additional folks in support and finance) is going to be working on it forever, but if you simplify it a bit you could keep 90% of it and only have 1% of engineering working on it, that might be a good tradeoff - after all, that leaves you more engineers to build features, which should drive additional sales. Unfortunately, that analysis never seems to get done up front and the cost is only understood after the billing system is deeply integrated into everything and would take an unpalatable amount of effort to change.
Yes, this is a big thing - there needs to be a clear model of how variable pricing and discounts are going to work in this company, with the sales team being able to apply the actual amounts of any prices and discounts, but not arbitrarily changing the model.
It doesn't work when the model is too simple or restrictive, which will simply result in the model being violated; you do need at least the ability to customize pricing for individual customers, and for specific invoices, but other than that the decisions (e.g. whether you apply invoice discounts as percentages or specific amount or both) can vary but you just need to pick any reasonable option and stick with it.
Yeah, this is tough because when the VP of Sales knows that one particular change will close the deal, they're going to push for it no matter what the cost. In reality it may be better from a big picture perspective to give a concession that is actually better for the customer and worse for the company than x as it relates to that particular deal, because the larger concession fits within the current billing system with no custom work, so the overall cost to the business is less.
Couldn't agree more on this. Even if finance, sales, marketing or other deps are involved, billing is an engineering thing! Back in 2016, while building the billing system for qonto.com (european fintech unicorn), we were surrounded by people willing to add complexity to a thing they don't understand.
A team of 2 engineers building it ended up in a whole squad called "Pricing Cross Functional Team"... even the name of this team was complex :)
Just wait until you meet billing's angry roommate: invoicing.
In the US, an invoice is just a weird PDF that you might glance at before sending off to your accounts payable team. But in other countries, especially those that use VAT style taxing systems, an invoice can be a legal document that expresses costs and taxes in a legally prescribed way for accounting purposes. Many countries have prescriptive rules over how you invoice, who can write invoices, when you can invoice, what goes on an invoice, etc. And every country does it differently.
Are you building a service that charges incrementally many times per period? Or even worse, refunds incrementally? You might be sending a truckload of expensive accounting papers to your customer every month, one per transaction. And each of those pages was printed using government prescribed software from oddball vendors you must use or else.
Many years ago I led a billing team at Yahoo! responsible for billing for premium services in European markets.
My team existed for the sole reason that the European business did not trust that the US payment services team understood European needs, and the "invoice is a legal document" thing was one of them. I spent so many meetings repeating to the US team that no, we could not switch to their invoicing as long as a new software release might retroactively change the template used to show already issued invoices (e.g. the registered office might change, or the VAT number).
We didn't have to deal with printed invoices thankfully, but we did ensure we produced each invoice once and stored it, so that the European finance team could sleep at night.
At the time Yahoo! had 8 different payment platforms. Some of those were due to weirdness around acquisitions and Japan (which was a joint venture), but apart from mine I believe two others also existed because of local weirdness that the local businesses decided it was safest not to let Yahoo! US mess with.
At the time I left, after 3 years of that, we'd managed to get agreements to migrate a few bits and pieces to the US after they'd shown the understood requirements for specific features, but it was like pulling teeth.
I'm Anh-Tho, one of a co-founders of Lago, thanks for your comment!
I was the country manager of France for a US Sequoia-backed company before, and I never ever managed to convince HQ to work on EU invoicing, I think they just did not want to open the pandora box you just described!
And even worse - your biggest customers won't even smell your invoice unless you enter it line by line into some ancient SAP system developed 20 years ago, where everything cloud related is classified as telephony except storage space which must be classified as filing cabinets (movable) or your invoice will be (very slowly) rejected.
And if it is rejected you have to enter it all again by hand; editing isn't a feature.
I don't get what you're talking about. Our biggest customer reluctantly agreed to accept preliminary invoices for acceptance over fax instead of signed-for mail, provided that we also ship a paper version with a red stamp promptly.
Wow, haven’t lived this pain fortunately… but totally understand how hard it is to try building a « modern pricing stack » but being forced to create invoices in old softwares like SAP…
This is a property of how VAT works. In the common scenario, the seller is effectively "tax administrator" for the buyer (much like employer pays payroll taxes on behalf of employee), therefore an invoice triggers two independent payables:
1. Seller to taxman
2. Buyer to seller
Neither seller cares how you handle your payments to the taxman, nor taxman cares about customer payments.
The second part regarding credit/debit invoices is again a property of "append-only double-entry bookkeeping". You fix wrong credit/debit entry with reversed debit/credit entry.
To spell that out: if you made a mistake in an invoice or negotiated a new total or an extra item (or had to drop an item) after creating the invoice, you now need to first create a correction invoice cancelling out the invoice you created and then create a new invoice. All three documents must have different numbers. Oh, and invoice numbers must be sequential and continuous.
Many years ago, my dad wrote an accounting system for the company he worked at to automate some of this. He spent ages trying to convince the tax authorities that the database was sufficient, and that there was no point in printing out an extra copy of every invoice and gluing it into a book. The idea is it'd make it harder for you to retroactively manipulate invoices.
They eventually saw the light, but it was a long slog.
In the recent years Germany has started allowing companies to no longer issue print invoices. But you still need to keep a copy of every generated invoice.
Initially "digital invoice" meant that you had to get the invoice cryptographically signed by the same government agency also in charge of literally printing money (or an officially licensed company) and then ideally send it using the now mostly defunct monstrosity that is De-Mail because it made guarantees about end-to-end encrpytion and sender/recipient authentication. Luckily this is now largely irrelevant and most companies just send regular PDFs via e-mail and/or make them available for download.
Yup, it's a nightmare. Invoices needing to be in specific number sequences. Any corrections need to be dealt with by reissuing a special "corrective invoice". Don't even get me started on geocoding for taxes. Urgh. Also, when you've got the invoicing right, you've got to work out how to do the feeds to the accounting systems so that it all posts to the right place.
Yep, fortunately in Europe we can issue credit notes, making our job easier. I do think giving the possibility to send charges information to a webhook is a great way to start: this data can be plugged to any invoicing system (or any tax, payment providers) who are trying to solve those things.
The hardest part for me, when working in that fintech, was to build the whole logic to trigger the invoice, not so much the invoice itself. But this company was a EU company, probably easier to work with invoices
I’ll add that the accounting side can get complicated really quickly. You end up with lots of different scenarios. Earned but unbilled: out of bundle call charges, for example. Billed but unearned: a paid in advance monthly fee, for example - this may need to be recognised as a percentage per day. Etc etc. So much to take into account when feeding to ledgers.
> When someone in sales issues an invoice with a negative amount
I worked briefly with a tax-saas, and to handle refunds I just... did a negative amount. It worked! Except.... I hadn't looked closely. Their API removed the negative. A charge of -$7.92 was just a charge of $7.92. Their docs indicated "we don't accept negative numbers - use a refund transaction instead" but... the API accepted the negative number, then changed it to a positive, and kept running. No error code returned. Really really poor experience, and I hope they've fixed that.
In Italy too. They specifically developed an XML format for that, and each time you issue an invoice you have to (well, with some exceptions, but they are gradually removing all of them) send a copy to the revenue agency, which will forward it to the recipient (and keep a copy).
While it is a bit inconvenient, though, I don't think it is a bad idea. I am pretty sure it helps a lot making the life harder for people evading taxes, and to some extent the availability of a standard machine-readable format makes it easy to do accounting. My wife sells videocourses online and I developed a thing that automatically issues invoices as soon as people buy a product (with some human supervision, mainly to avoid the machine doing havoc is some exceptional situation happens; normally it is just "click a button to send the invoice"). The invoices are then automatically available to the accountant for doing accountant things.
Not sure it is a good idea. It might be more flexible when you issue an invoice and have to modify it, but first it would enable fraud, and second it would mean that you have to monitor incoming invoices to see if anything changed and account for that. I think it's better to ask people to just emit correct invoices: if you need to invoice a bit more you issue another one, if you need to invoice less you issue a credit note. If you can't do that you need to work on your business processes.
And then just wait until you meet Invoicing's annoying cousin Purchase Orders (PO). THere is a PO. Can I pay this in multiple invoices ? Sorry the PO is not approved yet. We cannot accept an invoice. Sorry the Invoice doesn't have reference to our PO. Can you fix that please ?
Hey @codegeek! I'm Anh-Tho, one of the co-founders of Lago.
We're working on making the 'CPQ' : Configure Price Quote journey simpler, and this includes the PO dead end.
We're on a mission to make the pricing stack more open and customizable for companies (hence the open-source angle), and we're starting with the billing system.
Join the journey, we'll share our repo soon, and hopefully we can address these painpoints faster with the community contributions too!
Yep. Invoice software is country specific. One can sell invoicing software to several countries, it just can not be the same software. And no, you can't just abstract the differences, because the invoices focus changes widely.
At least we are in a situation where the seller is almost only subject to the laws of the seller's country. There are some exceptions, but one can mostly deal with those (the EU has plenty, the good news is that if the seller is not also on the EU, most do not apply).
Also, I haven't seen a country accounting system that made accounting papers inherently expensive. Some make refunds very expensive, but not the papers. It's more a matter of badly designed software and processes.
I work on a payment system that invoices consumers on behalf of other companies and pays out the money to these companies by splitting the payout on multiple partners (luckily I do not handle the support and follow-up). The system have ability to refund invoices and customers some times end up paying more than or just parts of the invoice. On top og all this it integrates with several old 90s systems with poor datetime handling and poor uptime (some times a windows xp box in the cudtomers offices). It also handles card payments and over all, by far the hardest thing to get right is the invoicing. Its just so extremely fuzzy and time sensitive.
And most big companies who agree to pay you net X days, that usually is after the presentation of a correct invoice. The invoice doesn't get to them, or it isn't right the clock doesn't start for them to pay you. Getting invoices wrong can quickly result in major cashflow drops.
I'm Anh-Tho, one of Lago's co-founders here.
Indeed, from our experience people see 'billing or invoicing' as a monolith block, whereas there's a whole 'pricing stack', billing > payment > invoicing > cash collection + accounting + CPQ + sales commission.
For cash collection, we mentioned the tool used by Lattice called Upflow which helps to manage account receivables.
Hi!
We decided to aggregate all costs of a "billable period".
Imagine you bill your customers monthly (the billable period), all the charges (usage-based features + subscription) will appear as line items of a single invoice.
This enables you to gather all the fees of a period into a total invoice, but still be able to provide granularity to your customers (breakdown of all the fees to be paid).
Hi, customer here. We'd like to change our billing period to start on the 31st of every month, and we'd like to pay 90 days in arrears, and we'd also like you to invoice us ahead of the billing month. Oh, and we're big enough that losing our business will end your startup, so don't get it wrong. Thanks!
Hey, Anh-Tho, just as a point of feedback, since you're leaving a lot of comments in this thread, it's not necessary to introduce yourself in each one. Most folks will be scanning the whole thread and seeing your introduction gets repetitive and breaks the conversational tone and makes it overly commercial. Thanks.
I was illustrating the point that customers are often unreasonable about billing, and there's no magical solution that fixes the problems. The answer is really to try to make sure you're not overly reliant on a single customer. That's hard as a startup though especially if you're in B2B.
Hey cm2012, I'm one of the cofounders of Lago (my business partner wrote this post).
Actually even if you're incorporated in the US, if you serve a client in EU for instance, the VAT rules of EU apply to how you're supposed to invoice your customer, beyond a certain transaction limit.
So... unless you're based in the US, and only serving US-based users, it gets complex very fast!
If you wire your dollars onto US soil, for a service performed in the US by US business, good luck getting the US legal system to enforce upon that US business whatever euro-cucked invoicing scheme is called for.
This doesn't matter. For B2B, if you don't show your EU customers a valid invoice, they simply cannot pay you.
It's not EU governments or post-purchase issues you have to deal, most of the time - it's trying to persuade a large accounting department to pay you unaccountable money. If you can't follow their invoicing requirements, it's just not going to happen.
Do people ever stop and wonder WHY in nations like Portugal the informal economy is roughly DOUBLE of say the US? When every invoice must be available to the government, what actually ends up happening is non-conforming invoices either get made up by the customer or the money magically flows out under some other auspice.
Making it literally illegal to accept an invoice that is legal in the country in which you obtained it borders on logic even my toddler can understand is absolutely begging for bad business climate or tax evasion.
What is the mechanism for French government to subpoena a non-EU business to find this invoice? If the customer (instead of vendor) produced an invoice that matched their bank statements and non-conforming receipt, hypothetically, would anyone really know the difference?
Probably they wouldn't as full audits aren't that common, but audits do happen and most accountants won't care about your purchase as much to personally commit a felony and forge documents for no good reason (accountants do have personal responsibility, and every accounting course reminds wannabe accountants that "the boss ordered it" is not an excuse) so they simply say that they can't/won't do it and if the vendor can't send a satisfactory invoice then they won't do business with that vendor.
How would a French audit uncover anything wrong with a conforming format invoice under the name of some random American company that exactly matched the bank statement, physical goods, and customs paperwork?
I don't think US has many relevant freedoms left. Some people cling very hard to legacy 18th century ones while the Moloch just gobbles up all the new ones stemming from progress.
That american exceptionalism approach will work with private citizen customers. With business customers, leaning back in the legal framework encouraging tax fraud will not fly. They just won't buy from you then.
That's why any US company wanting to do business in Europe either has an European subsidiary, or follows something that's compatible with EU invoicing schemes.
>. With business customers, leaning back in the legal framework encouraging tax fraud will not fly.
Yes which is why in nations like Portugal, where any business invoice may be subject to accountability to government, there totally isn't a massive informal (read: tax-evading) economy to deal with the fact that the government makes it literally impossible (at least in above words) to accept an invoice legal in the jurisdiction in which it was issued.
It's not illegal to pay an US company with an US invoice in the EU. (HN likes to invent problems where there aren't, and companies do that all the time). It is not even limited to the US, most other countries can't/won't follow the format.
However there might be extra steps in adding those invoices to their accounting. For example, when import, the buyer will be charged the import tax (instead of having it added to the invoice)
The informal economy of Portugal has nothing to do with the above btw since that has nothing to do with American invoices (and you can bet German rules are not much simpler) - not saying the bureaucracy isn't, most of the time, stupid. The US also has it fair share of stupid crap that people have to deal as well but it is "transparent" to most Americans.
You keep bringing this up and it's simply not true. The EU has a single set of VAT invoice requirements that applies across the EU (the MOSS invoicing standard). Any VAT invoice that satisfies these rules is legal in all EU countries (https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:32..., article 226).
My company does plenty of business with EU businesses and we use a single standard EU VAT invoice without issue.
It's not literally impossible - the simple solution to standard invoice being unacceptable is to issue a different invoice that matches the requirements, and almost every vendor is happy to do so in order to make the sale.
Also, the informal tax-evading economy relies on personal contacts and 'mutual understanding' - it's not easily accessible to a foreign vendor, and when foreign vendors do want to access that economy, it requires much more adoption of local customs than just making a different format of invoice.
Yes literally impossible to take something in a format legal in the US, that doesn't match the foreign requirements, based on European testimony above (the exact truth of which IDK).
Informal tax-evading (when the seller isn't evading, just the buyer) doesn't require personal contacts. This is just naïve thoughtless statement. It happens all the time. Example: business owner goes to Panama where they know no one. Buys a pallet of llama wool, seller gives a non-conforming receipt. Buyer goes back to France, imports the pallet as "cotton" and creates a fake invoice showing the Panamanian sold cotton. Seller then sells llama wool on the streets for cash, marking in the books that they sold "cotton" for significantly less. They then use the on-the-books "cotton" proceeds to buy a shit-wagon car or something, and fix it using the unrecorded cash on the side. They sell the now nice shit-wagon and record the proceeds as profit. Now all the money for the llama wool is accounted for.
Cross country tax-evading is only assured in this case to require personal contacts when the tax evading happens on _both_ sides. When it only happens on the EU side, there's no need for personal contacts on the US side.
> Buys a pallet of llama wool, seller gives a non-conforming receipt. Buyer goes back to France, imports the pallet as "cotton"
Customs catches him (distinguishing cotton from non-cotton is surprisingly easy, so is forged documents for invoicing and freight papers, especially given that Panama is an unusual country to export raw cotton - or raw wool - from), he is now in prison for 5 years for tax fraud and document forging, llama wool is confiscated and auctioned off to legitimate sellers.
Maybe you can come up with a better example then :) Here in the US pallets of cocaine regularly make it through customs. Our customs officers are generally one rung above high-school dropout. They'd generally be lucky to distinguish heroin from a pallet of sand, let alone cotton from wool.
Not in the export/import business myself, just passed through customs enough to see just how truly stupid and vapid these officers are, combined with the truly insane amount of import/export volume each one is responsible for. In the US distinguishing types of fabrics and ensuring various legal substances that look vaguely similar actually are what they say they are is near the bottom of the priorities list. I don't have a lot of experience with EU customs (except see last paragraph).
I know people want to believe customs is some highly intelligent caring group of individuals who are sleeplessly guarding the country and the purse in a carefully planned and executed manner with a fine-toothed comb. Anyone with this belief has probably never crossed an international border. In practice it's more like the mental equivalent of a 4 year old set loose with a completely unreliable "drug dog" that alerts on a grandma who is ruthlessly interrogated while one guy with llama wool and 10 pablo escabars pass on the other side.
I can recall one occasion where I stepped directly from fucking IRAQ into EU and not a single customs officer even looked at me. Not one. Just a stamp without a single question. I could have been carrying literally anything and no one cared. I could have been coming from Iraq for any reason and no one gave a fuck what that reason might be.
Yes. It would be even larger if US companies couldn't accept foreign invoices without being in a special format. Does tax evasion in the US somehow disprove tax evasion in Europe?
The difference is marginal at best. Most of the requirements are obvious and never/rarely change, like your address or the date. And adding a an incrementing number to a set of documents isn't exactly rocket science, either.
In this particular case, do you think non-US governments could protect the interests of consumers in their jurisdictions in a way that is less unfriendly to the businesses that want to sell to them?
In my experience invoicing, and really all 'printable' documents is the land of "oh someone did something wrong put X on the document". And then again and again and again... And "this is two pages long that's too much" and on and on ...
It's bike shedding insanity with no ideal or end in sight. Every change is arbitrary with no real measure of success.
I swear the only time anyone LOOKs at these documents is to bitch about them.
Oh man and just as if I summoned it someone just sent me a ticket about something on an invoice.
Just because you don't understand invoicing doesn't mean that the concerns that appear trivial to you are actually trivial.
From the perspective of someone who works with the people who process the invoices, putting "X on the document" and "two pages long" can be a huge deal when you have to deal with hundreds of invoices or when "X" can mean that an invoice must proceed down a different processing path. These aren't bike shedding concerns; they affect the actual processing of the invoice. In many cases, these concerns affect the validity of the invoice, meaning that the recipient's obligation to pay (or the associated deadline) is not triggered.
And doing something wrong on an invoice is never bikeshedding. An invoice is a legal contract, and in the E.U., an invoice is an unmodifiable document.
They're trivial... when you make the same change to add something, remove it, add it, remove it and hear that it's all being done "because X" and "X" never stops, it's clearly not working and the changes are trivial.
It sounds like you simply don't understand why they're doing it and you just assume that they're trivial because you have no knowledge of the domain.
Off the top of my head, I can thing of a dozen reasons for why they'd require something to be added, then removed, then added, etc., from one invoice to the next.
- Changes prompted by common vendor mistakes: DBA vs Legal vs Tax name, Tax vs business ID vs tax account id, projects vs Departments vs Accounts vs Budgets
- Changes prompted by compliance needs: business type (which governs the type and level of compliance required in most jurisdictions), changes to invoicing laws, changes to tax compliance laws
- Changes prompted by business needs: changes to the company's PO/invoicing/expensing process, changes to the company's financial, ERM, or payment processing software or service provider, one-off large expenses that aren't adequately handled by the existing data form; changes to financial modeling
- Changes prompted by marketing: new logo, design, or template
- Changes prompted by legal: new court rulings, etc., affecting legal's approval of the existing template(s)
- Changes prompted by employees: form is too complicated, form is not complicated enough, form doesn't capture my department or project's needs
The problem with an all inclusive form is that it is too complicated for most vendors and they end up filling it in wrong (requiring several rounds of review by the payor company), but too simple a form and you end up needing to make a lot of changes like the one the parent was talking about.
You could do multiple templates, but that just multiplies the work, and makes it more difficult for employees to figure out which template they should be using, and so they'll all end up using whatever random template they find first leading back to the One True Template and you're back to square one again.
And yes, all of the examples above are based on real life examples.
> You might be sending a truckload of expensive accounting papers to your customer every month, one per transaction. And each of those pages was printed using government prescribed software from oddball vendors you must use or else.
I guess that's good for employment numbers? There's really two ways to create jobs, innovate or regulate. The US and the EU have apparently chosen very different paths. With every country having it's own special snowflakes laws, there's a reason EU founders always try to come to the US first.
I work at a b2b company with a 25+ years history, and OH MY GOD...
* Sales is payed (partially) off commissions, so they do their very best to sell whatever the customer wants, not what the engineers know how to bill, or that product management has prepared. It's really hard to push back against a contract with revenue 10x your yearly salary "just" because the new features to bill it don't fit together with existing features.
* It's very hard to keep track of which customer pays for what, exactly, and is entitled to which service. Easy for standard products, hard for bespoke products, nearly impossible for shared services (it took us >9 months to sunset a shared HTTP proxy that was in use by just 4 customers that didn't even pay for it...)
* Legacy contracts: imagine 25 years iterating on product design, and keeping the old contracts running (except when the customer leaves on their own). Some of the old contracts are hard or impossible to find, nobody wants to spend their time rummaging through them to find out what exactly a customer is entitled to. A year ago we still had a customer that still pays for 8€ per GB HTTP traffic (web hosting), because that was the going rate back in the days
* Usage-based billing: half of our customers want to be flexible and only pay for what they use, and the other half uses SAP and for them, invoice amounts changing from month to month are the pure horror. So clever sales people invent hybrid models ("pay for what you use, but fixed price until a certain threshold, we'll notify you when you get close to reaching the threshold"). Another source of complexity.
* Usage-based billing: it's basically a topic for the whole company, most departments provide services that needs metering, but metering is usually an afterthought to the architecture, hard to verify for somebody on the outside
* Usage-based billing: for big customers you cannot put all the details on the invoice (our mailing service provider is limited to 99 pages per invoice...), so you need separate reporting to drill into the details. Fire and brimstone will rain down if reporting and invoice disagree on some numbers...
This just scratches the surface. Customers that want to split their bills along cost centers that don't align with access control tenants, different electricity prices based on location, the list is endless.
The article just scratches the surface too. I could have written a book to describe it!
Reporting for revenue is a hard game, indeed. Spent a lot of time trying to reconcile revenue streams in the same company we ended up building the whole billing system. It’s also not easy because metering is not something that is predictable nor easy to make it fit into a finance report.
Basically everything about enterprise is a nightmare for engineers. What amazes me (after having worked at some of the most profitable companies in the world) is just how little intelligence the leadership has about their own revenue or costs, beyond "wow huge amounts of money is coming in or going out". And how many critical processes are implemented manually, by individuals, with personal spreadsheets, on their laptops.
Yup, I work at a tech company and our entire finance & billing team is using excel exclusively (I think our payroll/leave system is the only thing not on excel), where every other team uses one system or another. All of the higher ups makes decisions based on docs created out of all those excel artifacts. I have to admit, their productivity is quite good and they are kings at aggregating and graphing data in sexy ways. It would be a challenge to build them something that actually works better and can adapt quickly to changes (the software lifecycle too slow for them and now they have to deal with some developer types to get something done, so skip all that noise and stick to excel).
It's a difficult topic to tackle; The financial leader has different needs than the tech one. One needs a perfect report with exact match day over day, month over month, year over year. The second one needs to do the math to calculate instant consumption of the product. If we add marketing leaders of a company to it, it brings creativity to the extreme (like inventing new prices, just to use it "as a hack"). Same goal (revenue), but definitely not the same path to success
> intelligence the leadership has about their own revenue or costs
I see the positionning of the CFO right next or below the CEO as a mechanism to let the CEO care about “big pitcure” money flow, while having someone in contact with reality guide decisions and veto stuff that won’t fly in cost/revenue terms.
I have a SaaS idea I’ve been sleeping in solely because building the billing, authentication, & marketing are each an additional 96% of the work that is not building the actual functionality of what I’d want to sell.
The reason billing systems are hard is because they tend to just accumulate features/behaviours/requirements. Forever. Although billing is not my specialty I have had to involve myself in it as product owner and business unit manager. Try and imagine the amount of logic, complexity, rules, and exceptions involved in a system that has accumulated perhaps 30-40 years of customer sign-ups. Umpteen contracts, umpteen iterations of each contract, and perhaps 50 or 100 products, each sold in 10 or 100 countries. I now know more about the tax system in South America than I ever wanted to.
I see building or implementing these kind of systems as "discovering" use cases for given scenarios/customers/markets. That's why "billing" is almost too generic - instead "billing for telco in South American markets" - is more useful because those verticals have specific sets of use-cases and challenges.
Yep, totally right. Same at the previous company we worked for and built the billing logic. You start doing it in house because the first version is simple. It’s getting complicated when you keep adding paying features to a product. Often see product managers becoming experts in taxes and accounting :)
> There was once a programmer who was attached to the court of the warlord of Wu. The warlord asked the programmer: ``Which is easier to design: an accounting package or an operating system?''
> ``An operating system,'' replied the programmer.
> The warlord uttered an exclamation of disbelief. ``Surely an accounting package is trivial next to the complexity of an operating system,'' he said.
> ``Not so,'' said the programmer, ``when designing an accounting package, the programmer operates as a mediator between people having different ideas: how it must operate, how its reports must appear, and how it must conform to the tax laws. By contrast, an operating system is not limited by outside appearances. When designing an operating system, the programmer seeks the simplest harmony between machine and ideas. This is why an operating system is easier to design.''
> The warlord of Wu nodded and smiled. ``That is all good and well, but which is easier to debug?''
Heh, I'd say the operating system is easier to debug, because the accounting package could literally have "bugs" that are caused by errors in the tax laws, business processes, etc, which cannot be fixed, only worked around.
Pro-rated billing is one of the many places outside of database query planners where Dynamic Programming is useful! Maybe there's a better way to do it, but well, if you give a man a hammer.. I wrote up how we did this at Equipmentshare back in 2016: https://tech.davis-hansson.com/p/price-wars.html
Billing is a nightmare, and if I had one piece of advice for people building a pay-for-what-you-use system like most SaaS, it's this:
DO NOT bill against your business logic entities. They change, and doing a COUNT at the end of the month won't catch all things which changed or which cost you money during the month. Instead, figure out what you bill for and and when you do that thing, record a row in a DB or into a steam of that "billable event".
Reconciling billable events is much easier to do, and it's tolerant of all the weird edge cases (such as a support person hard deleting data when they should soft delete, which would otherwise throw off your end of month counts).
There's a reason AWS (in general) can produce a log of everything you did which cost you money. It's painful, but it's less painful than the alternative.
Sound advice. Anything to do with accounting, you’ll probably want to treat as an append-only log of events. (Note, you can also use event-sourcing and have your domain entities _be_ events, in a billing bounded context that might make sense. Not usually the first approach I’d recommend though.)
On a similar note, make sure you think about bitemporality as well. In other words, “effective_at” vs. “”created_at”. You might learn that due to a bug, you didn’t record a billable event, and you need to insert it into the history (say, to put it in the correct billing period). But setting the “created_at” in the past brings a bunch of issues; it’s confusing and you’ll soon hit issues where lying about created-time trips you up. (Migrations, reasoning about what version of the code applied what events, etc).
If you squint, this is similar to how e-commerce billing is done. You have stock keeping units that represent the logical item being sold, and every transaction clones that row as part of the order record.
This way, if the price changes, the item description changes, the images on the item change, etc, you can still have a record of what was actually sold and how much to refund a return etc.
And depending on how you do it, it's not as hard as it sounds - for example, the first minute of "cloud server V1" can get a row in the db, and each "period" that row gets updated with the current minutes.
Or you can log them into tables that get processed into other invoice tables.
This also lets you keep grandfathered customers around as long as you want.
Couldn't agree more!
I do think the best way to do it is to log ingested events in a db, and decide which way you want to aggregate a billable feature at a defined billable period. This way lets you (i) keep track of everything, (ii) invoice complex behaviors and (iii) provide great granularity to your customers inside a final invoice.
I've had to deal with subscription based billing, and in my experience it wasn't a nightmare. There are some corner cases, but, those exist in every domain. And the stakes are definitely raised since you're dealing with people's money.
Usage based billing sounds like a nightmare though.
Taxes in the USA are a nightmare too. I have a friend that lives on a street, and for one side the tax rate is 7.5%, and the other side its 9.25%. Literally no company we tested got this right. Even Amazon.
For simple subscription-based billing, it's somehow not this hard, you are right. It also depends on how much plans you provide to your customers, and you still need to scratch your head for upgrades and downgrades.
I truly believe in usage-based billing, so I do think pricing are getting more and more complex over time.
Taxes are a nightmare for everyone also (in Europe too...)
Dont forget price changes. Gotta deal with that. Then discounts, trials, weird durations, and a bunch of other shit I forget.
My advice for anybody thinking of building their own subscription management system is to stop and go look at something like Zuora. You do NOT want to get stuck maintaining some home grown subscription management system--it will never be better than Zuora and you'll always be playing catchup, leaving your company unable to quickly react to the market. All the time spent adding functionality you can buy from a third party is money that you could have invested in your actual product. And worse, the fact you couldn't quickly react to the market because some key feature was missing in your homebrew thing is literally leaving money on the table.
Transitioning to any non-homegrown system is not trivial either, as you'll need some process to reconcile info from current system and some external system. However powerful some external systems are, planning a migration to something non-homegrown is itself a monstrous process. Benefits of staying 'in house' can simply be you're not wasting time trying to fit your existing processes and data in to something external.
That's really great that Ford uses Zuora. But for someone trying to work with an inhouse system that doesn't map to what a system like Zuora can handle... you now have technical work and a lot of political and non-technical work, and a lot of internal coordination to get all relevant people on board to commit to a change.
No company got it right - they all charged the same percentage regardless of which side of the street you lived on - so you didn't have a choice in the matter.
I implemented billing at WakaTime with Stripe, Braintree, and Coinbase as swappable payment providers. It's definitely a source of edge case bugs, but overall it works pretty well. It supports prorating subscriptions across providers, monthly & yearly billing, trails, discounts.
* We don't remit taxes for customers because that would be impossible to build in-house, but everything else was achievable.
* We have tons of logging, and generate transient Quotes on our end, then check that the expected charge matches what the payment provider actually charged.
* Each Quote is reproducible, so if any bugs come up we can give the "create quote" code the same input and see what went wrong.
* What helped when building was hourly tasks running in the background to make sure customers who should be billed were being charged, and make sure we didn't double charge any customers, and then alerts so we could find and fix those bugs.
I would do it again to support both PayPal and Stripe, but it's definitely a source of complexity for the code base.
Hi!
I co-led the billing stack re-write at Uber circa 2017 and it was an absolute nightmare.
Some of the unexpected hard things that are not mentioned in this article:
- Manage promotion was very complex
- Modeling money (we used to model it as cents ie 1/100 of the common denomination ... turns out it broke a lot of countries like Japan where the Japanese Yen doesn't have cents)
- Timezone is a hellhole
One design decision that, for me, seems to simplify things is to consider the "business system" a type of state machine that records all business events and serves as a "source of truth". If the events are not recorded, they have not occurred from the business perspective. A ledger-type architecture can be useful.
This means that business events or user operations generate state transitions, which eventually are implemented as database transactions. The event log can be stored and inspected.
The end user terms are encoded in the state-transitions. Contract obligations are encoded in the state of the database.
As for calendar issues - operations need to be performed over "real calendars". It might be practical to "materialize" the business calendar into discrete units for this purpose.
For sure; I do think an event based solution working as a source of truth for the billing is the right solution. However it still creates engineering difficulties (making sure you don't ingest the event twice for instance). The ledger-type architecture can definitely work. When we built the system for a fintech, it was actually an event-based architecture connected to a ledger (taking the money out of a wallet).
I think the whole process would be:
- Ingesting events for usage based feature
- Store these events in a database and make sure you make the idempotency right
- Aggregate the usage of these events based on most common aggregate functions
- Price this usage inside plans or subscriptions
- Assign a subscription to a customer
- Trigger a webhook (used for invoice/payment) at the end or beginning of the billing period
And that events, as much as possible, are formulated as observations or state-changes rather than operations. Events in the form of "set x=1" or "y=0 was observed" are idempotent, where as "do this with that" is problematic.
In the framework of state-transitions, actions such as "price this with that" are implicit in the state matrix, and shouldn't be called from APIs. Rather, API calls should be used to produce reports which are print-outs of the "current state". Webhooks in this framework are attempted state-transitions. User interactions generate events, and requests for reports (for example invoices), but as little as possible in the form "do this with that".
My opinion currently is that there is little value in sharing code (libraries/frameworks) for these sorts of solutions because the code is trivial, application/language/domain-dependent, and coupled to everything else in the system.
But design patterns, templates, and concise examples are hugely valuable because they help illustrate the complexity and solutions up front.
I was reading this and thought event based system for generating the billable quantities. I am sure billing by api call or data consumption must be done by an event driven system, if my customers are doing a billion requests a month, I probably do not want that data in my database. I would have also thought logging would be your friend in generating reports to reconcile your accounts. Some form of message cache to ensure it has not been processed twice would help.
Nit: there's a lot of noise being made in comments about it being an open source product, but I haven't found the source (maybe I have to speak to an expert to get it?). github.com/getlago has the docs, which is nice (and gives a lot more info than is available on the website about how the product works), but not quite the same.
Also note:
> Our open-core version is forever free. We will introduce paying add-ons in the future, with a consumption-based approach.
Awesome, glad to hear, would love to take a look whenever it's available! Looking at the docs it seems like the current state of the product is much more about subscriptions than usage based billing (e.g. no aggregation or rating). Any plans to add those in?
We definitely want to address usage-based pricings, but also all the pricings that are hybrid between usage and subscriptions (ie the vast majority of software pricings). Feel free to reach out if you have more question: anhtho@getlago.com
I spent a lot of time right out of college working on the equivalent of an invoicing system for a pharmacy chain (rx drug pricing, electronic carrier submission, and reconciliation).
Account receivables is also another nightmare. We would get checks that randomly show up at corporate for no reason from insurance carriers but then when the carrier realized their error instead of handling it as a separate process they would deduct the check amount from whatever invoice ( or even across a range of invoices! ) from us they felt like. We literally had a bucket called "magic money" these random deposits went into that we would use to fill in the A/R gaps from insurance carrier insanity. There was no connection between magic money and whatever invoice they decided to short change us on so it was just a hope-for-the-best process.
Hey,
I feel your pain. The "hope-for-the-best process" definitely is something happening in many companies...
Yet, it's not a fatality anymore. I work with Upflow (YC 20) and it is exactly what we offer to our clients : 1. Making sure they have clear visibility on their AR 2. designing systematized workflows for their customers (as many as needed) 3. Customer payment portals to help their customers pay on time, with no friction.
That's here at www.upflow.io
I've implemented a few billing systems and "the nightmares" were never related toengineering but always to specification: If bizdev does not know what they want, no outsourced imlementation will be able to help them.
From your experience, who should specify the billing system? The Product team?
We're building Lago so that the right questions / decisions frameworks are asked during the implementation, so it's like a forcing power embedded in the product.
Our experience is, unless the Product/Biz team has been exposed to billing, they will never specify in a way that is precise enough, so the engineers who implement billing will have to think/assess/decide themselves.
Have someone from accounting/legal work at least 50% in bizdev; don't let them talk to engineering until they have sorted out what they want and, much more important, whether it is legal..
But equally important is to cut back engineering's enthusiasm for abstraction/unification": In my experience it is better to have 2 distinct, tested implementations against one pretty generic data model than trying to coerce both into one. Intellectually one can, but "oh so great billing concept" #3 or #4 will break ones neck.
I don't really think anything about billing is hard, it is just incredibly dull if done the pragmatic way.
'Tax logic' is an oxymoron and a trap for smart technologists to fall into. Tax codes are not logically consistent or clearly defined. They're a bunch of illfitting laws, beauracratic/regulatory guidance and interpretations of judicial decisions.
I tell folks working in this area to accept the illogical nature of it all and to be ready for all sorts of arbitrary last minute changes. As the article points out understanding time plays an important role here too - when do different tax treatments take effect is an important question to answer.
Tax is written by a bunch of people whose background is overwhelmingly from law, not finance, for reasons that have nothing to do with logic, consistency, financial literacy, understanding, sanity, or common sense. Politicians frequently pass laws without understanding what it will do or what its implications are, because they just didn't think that far ahead. It is quite possibly the worst system you could come up with for writing tax law.
That said, there is a logic to most of it, but not in a way that allows you to come up with general heuristics or universal abstractions.
Any kind of business or accounting system tends to be an endless long tail of "can the invoicing system send the invoice in Esperanto using only 16-bit Unicode characters on Tuesdays but only while it is raining and only for customers whose last names end with E and who signed up more than one year ago according to the Chinese calendar?" type features.
That's why attempts at "no-code" systems where biz/accounting people can design their own system are a perennial fad, but as these systems are developed they always just end up evolving into weird domain specific programming languages.
It's also why any sufficiently large company ends up implementing their own in-house accounting system and/or customizing some byzantine monstrosity like SAP which is really just another way of implementing your own.
The #1 reason those systems are moderately successful: When your sales person needs to write the logic for billing customer X, they'll try to make the customer contract more reasonable.
Normally sales people are deal-driven, and if promising the customer "14% off on the first three transactions on every rainy Tuesday, unless Monday was also rainy" gets the deal closed, then that's going in the contract.
No better way to ensure the contracts are reasonable, than making sure the person negotiating the deal feels the pain of billing it
Agreed! this triggers also complex behaviors because sales people want to handle this in their env (Salesforce, hubspot or whatever); You also have to build the logic to update subscriptions and pricing directly from it, in a two-way sync. Things getting crazier ;)
Just wait until you have to do this under government controlled (tariff) environments. Years ago I worked on a tele-conference platform for large international telecom. We built the software to run all aspects of their platform including billing.
Government tariffs covered billing rules for each leg of a call including rates based on caller's local time of day. Converting bridge time to local time via area/country codes(only data available) and time-zones is quite a mess. Add DST calculations to that per locale to reconcile negative duration legs or legs that should be 1.5 hours but show up as .5 hours just makes it cloudier.
If your system repeatedly gets it wrong, you could be in trouble for billing fraud or tariff violation. No joy in that job.
I agree with thesis but there are entire aspects of this that he didn't even touch upon. There is a multiverse of different billing nightmares for engineers.
I worked on a piece of software that bridged billing data from IBM mainframes out to the web. The hoops that had to be jumped through to get data out of the mainframes and the number of hacks and kludges involved was legendary. Likely everyone here has paid many bills in their life on websites that used that software. At one point I knew the vast majority of my personal bills had gone through it. Credit Cards, Utilities, Gas cards, etc.. Stuff everyone had to pay with paper before the web.
Basically the mainframes were almost incapable of any data exchange that was meaningful in the 21st century. Instead of IBM doing something same the market just built software to scrape bills off the mainframes printer output, which you could intercept in electronic form. But then every accounts bill files were different. And sometimes the mainframes moved data around on the page, and so on and so forth.
That gives me flashbacks. I worked on a payment system that had to pull data from third-party service providers that only made it available in the form of downloadable spreadsheets from godawful slow and unreliable JavaScript-heavy web portals. Just getting to the point of downloading the spreadsheets was painful enough, and then the spreadsheets themselves were inconsistent and would sometimes flap back and forth between two different formats for a few weeks as the service provider repeatedly deployed and then rolled back new versions of their software.
Always has been. Marketing and sales create complexity on the front end, devising all kinds of products, plans, discount schemes and related complexity; jurisdiction, taxes, and accounting rules add complexity on the backend. In the middle, measuring what you're billing for is the hook around which all that complexity chaotically orbits.
Decades ago - the computer involved was an honest to goodness System 360 (which is to say, not a 370, Z, or any of the the code compatible 360 successors) - I spent an evening trying to figure out the service charge on my bank account. I couldn't get at all close to the figure on the statement by applying the bank's schedule of charges. I took it into the bank, and got bumped by the teller to a department manager, and thence to a VP (small town; small bank), whom I told, "I will pay the service charge if you can explain to me what I'm being charged for." 2 hours later he gave up, having not gotten any where near an explanation, grinned, and said, "I can fix this." He set some flag on my account, and I never again paid service charges at that bank. Yep - that flag remained set on my account for nearly 20 years, through multiple software and hardware upgrades, until the bank was acquired by a big, expanding regional bank, and I left for a different, entirely local bank.
The marketing team of my previous company created a whole fake price plan that never existed in the code base. Somehow it works to drag acquisition, but it’s a pain in the ** to explain why it’s not as easy as updating webflow ;)
As a co/founder and core architect of multiple PCI level 1 compliant payment processing companies in the last two decades I will add as others have that billing and payments is exactly like anything else which is "difficult". Once you spend enough time in the trenches dealing with the challenging technical issues your experience and creativity leads to the proper solutions that further enhance the system. Everything is just inputs and outputs yet it is what one does in between those "puts" which matters and thus where experience can reduce such challenges.
It's hard to figure out what it can do.
Everywhere is their branding.
As I understand they just send payment receipts, but no invoices.
I'd like to offer a free trial. While they have support for this, a user needs to enter his credit card right at sign up for trials to work.
So in the end I have to build a hybrid solution where my system gets tied to theirs. Still I don't have full control over all e-mails that are sent to the user. I'm not 100% happy, but better than not charging money.
Stripe is fairly straight to set up if you sell products in the US and only accept cards.
For selling a SaaS worldwide, Stripe quickly becomes just as much of a nightmare. Taxes, billing, chargebacks, subscriptions, web hooks, different payment methods, VAT validation, card expiries, legislation, changing laws, SCA flows, and my favourite: SCA iframes that become non-responsive on browsers in a certain language. It is a never ending source of edge cases.
Don't ask me how I know... My biggest mistake with my business was that I vastly underestimated how much work it is to accept payments. In my case, it proved to be more work than building the product itself.
Exactly that. While stripe would tick my needs, it's gonna cause some headaches on the long run. They can calculate the taxes meanwhile, but you have to still take care of e.g. MOSS yourself.
I'm the CTO of a company tackling exactly this problem (www.salesbricks.com) and I'd go one further.
The complexity in billing, invoicing, payments etc are _not_ the stuff that computers or systems do. The complexity is and always will be in the stuff humans do.
Don't spend time, effort and money on things that do not bring business value. Any time I see a company with engineers building something they could buy off the shelf [and the thing they're building is not the product] I know their priorities are whack. I've worked for several companies that had more money than brains. They'd spend years, and millions of dollars in salaries, to build shit they could have bought and implemented in two months. And not only that, but multiple iterations of failing to build said thing, by multiple teams.
Just because a tractor is used for farming doesn't mean a farmer should build her own tractor.
> Is there an off-the-shelf billing product available out there?
There are lots and lots of OTS billing systems. They might work the way you want or need. Billing is the classic "I wanted wheel, except I wanted a round one" kind of problem. Right now, I'm looking for a system that will let sales invoice whatever they put on paper, and be good a structured, recurring billing that is easy to hook up to my SaaS platform. Oh, and Bob in sales wants to be able to edit the invoices after they are issued...
I rewrote an invoicing system (originally in .NET) in Python and yeah, I totally feel this. All the system did was generate a CSV file that could be input into another system (Quickbooks, iirc). The original .NET version was nonsensical at times, replete with bugs, and I spent a lot of time trying to figure out what the original intent was (there were no docs, of course, and finance often had conflicting/incorrect ideas about what the system should do.) Version 1 had to simply replicate the csv generated by the original system exactly, which required purposefully writing some bugs that I discovered along the way.
But yeah, there were all sorts of nasty complexities -- this was a usage based system, but there were (optional) tiers (pay less for higher tiers), as well as certain time segments that were discounted. And then there were cases where multiple accounts could share a set of data and get a discounted rate and/or only one of the accounts would be billed. And on and on....
One lesson I drew from this experience was that, when discussing things with finance to try to figure out what the requirements were, it really helped to have pictures. I would create these elaborate usage/time graphs, and color in portions to reflect different billing/pricing scenarios. These made it much easier to discuss the current/future behavior of the system. Because saying "really, this is simply the integral of a usage function over time" didn't fly.
Not the OP, but anybody who has built a deep integration with any billing provider (inclusive of Stripe) knows there are some bodies buried, although you typically don't discover them until you're late in the game.
90% of your use cases are covered, and with excellent UX and docs to boot. But that sweet 10%!
---
For a concrete example, Stripe has a very nice "subscription" abstraction that makes it easier to operate a SaaS business. They also offer a nice "tax" product, that doesn't do filing for you, but does offer tax computation if you code your products appropriately. Sounds great!
But if you want to sell 2 subscriptions to the same customer, and those 2 subscriptions are fulfilled in 2 distinct taxable regimes (say, you're shipping widgets to NYC once per month and to a vacation home in NH once per quarter), you're screwed because Stripe only allows 1 shipping address for customer for tax purposes, and each subscription references this address when computing tax.
We went through support, which filtered through to engineering, and the eventual answer was "no plan to change this anytime soon", which is difficult to interpret any other way than "good luck". :)
There is no clean or even acceptably-dirty solution to this problem, tragically. So as a business we can't allow customers to transact with us this way. Oh well.
There's another, similar, story around creating draft invoices to give people quotes before checking out. The dashboard API has a function to modify the line items of a draft invoice, but it's not a public-facing API and there's no short-term intention to make it public-facing, so one has to build an ugly workaround.
This effectively means one needs to maintain internal models for subscriptions, products, taxes, invoices, invoice items, products, shipping addresses, on and on. And Stripe becomes a somewhat-dumb processor for these models (although the state machine for subscriptions is admittedly very valuable).
I'd love to "offshore" everything to Stripe, status as the source of truth for data included, but it's not feasible in a surprising frequency of scenarios.
Caveat that every payment provider is like this, Stripe is much better and actually improves over time.
A lot of the time there’s trade-offs that can be made that engineers don’t even consider because they’re not pure. For example, why do you need a one-to-one relationship between your customers and Stripe customers? I am guilty of it myself, where an 80% solution is unacceptable from an engineering perspective but, actually, when balanced against the cost of building out a perfect solution, the business may well find it very acceptable.
One of the most important things I’ve learned about working with third-party systems is that there has to be some amount of surrender to their way of doing things. Essentially, the price of using Stripe is not being able to use invoices as a quote, and that has to be articulated to the business. If that’s not acceptable, then the business should be considering the requirements in aggregate when evaluating a solution, rather than deciding on Stripe and then gradually re-implementing parts of Stripe internally because they aren’t right for the business.
(Obviously that’s easier said than done, and for engineers it’s a fun challenge to make something impossible possible, but it’s almost always a mistake and as engineers it’s our responsibility to articulate the burden of owning workarounds.)
The example you give is a smart one that we did explore!
Effectively, relationship is no longer user -> stripe customer, but user -> shipping_address -> stripe customer.
We decided the cost of implementing and maintaining this solution wasn't worth the benefit to the business. A credit to your later points.
We came up with a solution that allows us to ship one-off things to arbitrary addresses with correct tax calculation, and if a customer really wanted to setup a subscription to a distinct address, we could implement it in the future, hackily, with this primitive.
>One of the most important things I’ve learned about working with third-party systems is that there has to be some amount of surrender to their way of doing things. Essentially, the price of using Stripe is not being able to use invoices as a quote, and that has to be articulated to the business.
We have this same limitation on another payment platform. We just change the concept on our end to "payers" and allow an account to have multiple payers. When the user adds a payment method, they can either select that existing payer or, if they need to change the name or address, they create a new one.
The provider we use had an invoicing product, but it has several misfeatures that just make the product unworkable. We absolutely have to be able to enforce first-in-first-out invoice payments. Users will pay the May invoice before the April invoice and pretend like they are all paid up. I'm not sure why they didn't consider this with a subscription-invoice product... users paying the invoices out of order was a big problem because users saw it as an exploitable vulnerability they could use to get a free month here and there.
> There's another, similar, story around creating draft invoices to give people quotes before checking out. The dashboard API has a function to modify the line items of a draft invoice, but it's not a public-facing API and there's no short-term intention to make it public-facing, so one has to build an ugly workaround.
There is a similar "not a public-facing API" limitation with configuring multiple invoice email addresses for Stripe customers (works in the dashboard, but blocked in the public API, where you can only set a single email per customer).
I'm actually quite surprised Stripe does this, since their initial reason for being was developer-friendly payments. But oh well, maybe I should take a look at an alternative (like Lago) sometime.
We're working on making the quoting process simpler indeed.
What we've learnt that the only companies that can afford a proper quoting system are enterprises: they implement Salesforce CPQ (configure price quote) or Zuora. They are mostly sales-driven, so the investment is worth it.
But it does not make sense for PLG companies, who start with self-serve and layer sales afterwards: the investment is not worth it, and it creates 2 different systems: self-serve and sales-driven billings, not great for the user experience (and for the internal processes!)
I think it's easier for billing systems to have 1 subscription for 1 customer. The reason behind this is mainly because you can apply several prices to the same feature belonging to two different subscriptions. Stripe does not work on it because they might consider this as a edge case.
When you look at usage-based billing, even if a billed feature is linked to consumption, it needs to be related to a subscription given the fact that this feature can be priced differently for several plans. Every company has a singular pricing model in mind and it's hard to have an all-in-one model fitting all the needs.
Even when we built internally the whole system for a fintech back in 2016, we had some edgecases creating conflict inside the company.
But I'd push back on the 1 subscription per customer idea. Consider any physical fulfillment business, where you have an optional annual membership (amazon prime, costco, whatever) and N subscriptions of varying frequencies for physical products.
It doesn't make sense to roll these into a single subscription on the backend (unless the periodicity happens to line up perfectly, and even then there's a heated debate, since you still probably want them to be separate charges).
You can go down the route of creating N "virtual" customers for each "real" customer, but it doesn't really solve the problems, because these N customers have a complex relationship with each other that you need to encode and manage.
Hence, the name of this post :). It's definitely not easy!
This is what I’m doing, but then you don’t get a clean history in Stripe, which is annoying to folks (e.g. to our support team). So, it’s something I’m thinking about changing.
Pretty much everything is more complicated than people (including engineers) think.
Take a simple example: selling clothing in a retail store. Sounds simple right? You have an inventory of items and someone pays for it. Should be a simple transaction, right? Well, now you're dealing with:
- How do you determine the price? There might be a retail price but there are sales, discounts, overrides for various reasons (eg defective goods), etc;
- You have to handle different payment options eg cash, credit card, debit card, gift cards, store credit. Payment methods can be mixed;
- You have to handle returns. These generally have to go back to the original form of payment. Some goods might not be eligible for a return. What if the payment is split between store credit and a credit card? Where does the balance go?
- Identity of the person making the sale or handling the return;
- Authorization procedures for returns and price overrides.
Everything is complicated.
As an aside, this is one reason why I'm so bearish on Web3 and smart contracts in general. The edge cases are so complex and possibly unknowable that codifying these, particularly on an immutable blockchain, to remove the need for human intervention seems doomed to failure.
Take Web3 identity. If you lose your password there are methods for recovering your account. The first is a reset password option. That may be insufficient and there'll sometimes be a human avenue to recover your account. Now consider a Web3 identity. I've seen various incarnations of these such as authorizing other identities who with consensus can recover your account. Well, those identities can be compromises so you've just added an attack vector. Alterantively that recovery mechanism may become insufficient as people lose identities themselves.
If these problems weren't complex they'd be solved problems.
> As an aside, this is one reason why I'm so bearish on Web3 and smart contracts in general. The edge cases are so complex and possibly unknowable that codifying these, particularly on an immutable blockchain, to remove the need for human intervention seems doomed to failure.
I do think the potential for errors is currently real (people mess up simple wallet transfers all of the time) but it's reasonable to assume that big things and making them easy for normal people to use is going to take time.
It's easy to forget how ridiculously early into crypto humans still are. We're only about 14 years into the invention of crypto. We're only about 7 years since the release of Ethereum. The general public barely understands what crypto is other than having some conception of it as some kind of Internet money. Most people cannot discuss what a blockchain is, what smart contracts are, or why oracles are important. And forget the general public for a minute: even smart technical people on HN can barely give a good description of how all of these pieces work and fit together.
>If these problems weren't complex they'd be solved problems.
That's an axiomatic and therefor meaningless statement. Knowing something is hard to solve doesn't mean new tech won't bring improvements regarding the problem.
It doesn't follow that having an immutable ledger means it has to be used for every single application. That would be like claiming trains are useless because they can't climb stairs. They don't have to do that, we keep walking the stairs just like we've always done.
Immutability is great in situations where you want to record that something has taken place and don't want disputes about this. Some of the things you mentioned can be automated to some degree. Take returns as an example: You sell the item with a unique token that makes it identifiable and everything else can be automated, including chargebacks. We already have this to some extent with bar codes and RFID tags, but it can be taken to the next level with a non-fungible and non-falsifiable token that's connected to a smart contract which automatically returns money, does the accounting, updates the inventory, reports the tax data and so forth if authorized by the right person. As the owner you can easier monitor and analyze the flow of money, better detect fraud and come up with a myriad of improvements. Let's assume we have autonomous delivery trucks soon, then the entire supply process could be automated, including the billing as soon as the product is accepted. With something like the Helium Network shipments can be tracked, so there can be no dispute over where they are located.
For company owners who really doesn't want to spend much time at the office you could go a step further and even set conditions for disputes like contracts triggering a certain action after a suppliers products have been rejected too many times because of quality issues, or vice versa is a customer keeps ordering and returning products from a factory. The contract could switch to automatically ordering from a competitor.
Hey Cletus! I'm Anh-Tho, one of the cofounders of Lago.
Totally agree with you on web3 identity and its edge cases, it's just billing ^10000 in terms of complexity.
I think for 'billing', usually people (especially non-tech people), don't understand the different blocks of the pricing stack: authorization, pricing grid, billing itself, invoicing (they often only see end-result: a receipt or invoice), payment gateways, dunnings, accounting etc).
This post barely scratches the surface of it, but we intend to write and deep-dive more on each of these building blocks.
A relational database scalable for the billing data volume, should be a best answer, in term of realtime process and analysis，say complex query.
Suggest an article of a big logistics company case here, not only billing, but also order, messaging, transshipment:
https://en.pingcap.com/blog/how-to-reduce-cost-per-order-wit...
Anyone know of a company other than Paddle and PayPro Global that act as the merchant of record and remit taxes for you globally?
I'm a solo founder not interested in dealing with either the complexities of these billing issues nor with the taxes issue. But my product would easily be of interest globally and the API's I've built have more interenational customers than global customers so I suspect the same may be true for the product I'm building out of those apis...
Poked around their site a bit, they claim to be open source but I don't see a link to a git repo anywhere. Also searched GitHub a bit but might have missed it.
Hey @daxaxelrod! I'm one of the cofounders of Lago, the lib will be opened very soon, we're in the final steps of QA, that's why. I will make sure to ping you back here when it is.
We wanted to share this post about our first-hand experience with billing a bit ahead of this.
Thanks for your patience!
> Why billing systems are a nightmare for engineers
My guess: Because basically nobody wants to develop one of these as a hobby FOSS project. The whole idea is collecting people's money, which when you write your own code for public benefit, you basically don't do :-)
> We quickly decided to add plans, and 'pay-as-you-go' features
So, the author talks about things getting more complex, beyond their expectations. But - why did they not expect this? I mean, they know that companies pricing changes over time; that payment models change, including rates, cadence, use of internal credit etc. We all know this without ever having worked on a billing system.
> Qonto is a 'neobank' ... Fintech unicorn
So they were expecting billing to be easy for a bank-like endeavor? Hmm.
> #1 - Dates
The date problems per se are quite mundane and any non-trivial software system which involves timed activity deals with them. The billion period problems are not really date problems and are a recap of what OP had already written about. They are also problems most of us are well aware of from our personal experience as people in modern society ("Does my bus pass expire a day after I bought it, i.e. 24 hours later, or does it expire at the turn of the day = at midnight?")
>/as Qonto is a 'neobank', we wanted to charge our customers directly in their wallet, through a ledger connected to our internal billing system. The team started from a duo of full-time engineers building a billing system (which is already a considerable investment), to currently a dedicated cross-functional team called 'pricing'.
Stored procedures (ca. Oracle v6) suddenly sound a walk in the park.
Depends on your requirements I would say. If your target customers are global, then yes, you'll have to deal with a lot of rules and regulations. But the engineering side of things I don't think is that hard.
Perhaps when you first enter the domain it seems like a lot because you are unfamiliar with it, but such is it with all things.
Also, these guys have an economical incentive of convincing you it's hard. So keep that in mind.
B2B SaaS founder here, several years in: this article is right on point. Additionally, using Stripe Billing is not all it's cracked up to be, as it actually doesn't solve all my problems, while taking a chunk of revenue.
I've been building parts of a billing system myself over the years and I'm considering removing the remaining external parts, because I'm not happy with their limitations (Braintree's subscriptions are really limited, and absolutely nobody does correct invoicing for my situation (EU/Poland)).
Also, in a B2B SaaS you will end up with customers on totally different payment methods. Not every business has or wants to use a credit card. So you get annual billing with proforma invoices and the complexities involved with switching from invoiced billing to credit card billing and vice versa.
I think companies should not try to reinvent the wheel with building billing systems. It is better to implement an ERP to take care of billing rather than building out your own. If you ever grow large enough, this homegrown system will never be SOX compliant and will not stand up to audits.
> Another option is to rely on existing billing platforms, built by specialized teams. If you’re considering choosing one or switching, and you think I can help, please reach out!
To note, this option only absolve of _some_ of the complexity and burden. You still have to understand:
- what problems the billing platforms is solving for you, how they're solving it (does it match the process you need ? -> you also need to know how it should work) and what to provide them to make it work.
- what problems they are _not_ solving, and you need to deal with on your end
A good service provider will help you at least on the first part, and educate on the parts you need to be aware of, but there's always dark corners that easily get overlooked if the business owner didn't do their homework.
Reading this article gives me a smidgen of sympathy for Xfinity. They have so many plan options/timed promotional deals, and seem to run into so many issues every seemingly minor change, now it makes sense to me why.
Anh-Tho, one of the co-founders of Lago, here!
yes totally, each time I open a pricing page, I have full sympathy for the engineering team who made it happen! I used to be the one tweaking with pricing plans on a spreadsheet and handing it over to engineers in a 'just make it happen' fashion!
Problem is it still requires an API to integrate. Stripe has all these capabilities but API integrations are needed to use them. If you are looking to build a micro SaaS without needing to deal with billing API integrations check out our software: https://saasbox.net. Built for completely eliminating any billing related SW development. It doesn't handle all the corner cases mentioned in the article, but some of them are handled, such as plan upgrade / downgrades with pro-rating, editing plans on the fly, migrating users across plans, notifying your application on those changes.
My understanding is that Stripe billing can be managed completely no-code from the Stripe dashboard? And I think it includes automatic tax, prorations etc.
My very first startup (wayyyy back in 2000) built a billing SaaS. I fully agree that rating, billing, invoicing, and all of the related things are hard. I learned a lot in that role about dealing with complexity...and date/time.
Wow, that's a tough one. Just spitballing, I'd say:
1). Don't try to build it yourself, no matter how simple your case. Your needs will ALWAYS evolve to be more complex AND keeping up with the thousands (or more) of individual regulatory regimes in the world is a massive task.
2). At any given point in time, you need to know the state of everything required to rate, bill, and invoice at any other point in time. (i.e. bitemporal). As a consequence, critical data effectively becomes immutable, auditing is possible, and allows pre/post views of things after corrections.
3). Once you start to get into tiered/volume discount, combo discounts, providing multiple services, etc. complexity rapidly increases, so observability into how exactly things are calculated is of paramount importance. This one is probably pretty obvious, but missed a lot. How many times have you received a bill with little to now information about how things were calculated? It's possibly a business decision, but just as likely the process is a black box and not readily observable.
I just want to say that this article is a great intro to the topic and I credit a lot of my growth and development as an engineer to working on billing systems in my first job out of college.
Don't build a billing system. Or at least not a precise one.
Have a price list as normal, but just guestimate the user's bills based on any easily accessible information.
Eg. For a hosting company I might just count how many VM's they have active when the billing script runs . If the user isn't happy with the total, give them a button to correct the total and pay that.
Spot check what the user's pay, and warn then ban any users which are clearly deliberately paying substantially less than list price for your services.
Someone's going to reverse engineer your heuristic and min-max it. At that point, you'll need to defend your heuristic against edge cases. And then you may as well just make it official.
Also, B2B customers are allergic to "pay what you feel"-type schemes.
You’d still have to apply the right tax rate for every line item. If you allow the user to edit the total, then which of the tax amounts are you going to recalculate?
As someone, who has built a billing system not once in my life, but twice (one for an internet privider I worked for, which counted the amount of traffic, and another one for a SaaS project(, I fully sympathize with the post. Billing is an unbelievable can of worms even before you get to taxes. Add in all the things the marketing people want from billing (trials, discounts, per-seat per usage pricing, etc), and you have enough tasks to last till retirement, no matter how young you are.
Yep, do agree that non tech team created a lot of pricing complexities, sometimes without any reasons. Remember a marketing team updating plans on webflow and telling the product team « see, not that hard » ;)
Ended up with a whole squad of engineers called « the pricing cross functional team » …
Billing is a nightmare for everyone but its not so bad for engineers. In general it's great to keep track of what things cost but in situations where the overhead is more expensive than the product it self we should be considering more unmetered approaches.
Billing, like taxes, can be broken down into relatively simple smaller problems. You solve all of those then do a bit of maintenance. In return you get something close enough to all-you-can eat on the cheap! Automation is wonderful!
Great article and excited to see another solution tackling the complex problem of billing. I’m an founding engineer at https://metronome.com (building usage-based billing infrastructure) and definitely echo the sentiments shared here — building a billing system that can scale is no small feat and anyone who previously built billing in-house can attest to just how painful that is.
Can corroborate, my experience working on an upgrade to an existing multi-platform (iOS IAP + Strip for web product) subscription service was certainly a bit painful.
Yes, upgrades and downgrades were painful for me too. I think it's due to different subscription dates between all our customers, having to create specific code for each edge case. In the end, it works, but so much harder than I thought
Usage based billing is not only hard to implement but hard to consume. No one really knows what they are spending and how much particular customers or products cost. I did a writeup a week ago on CDNs and ended up spending several hours in spreadsheets.
I had felt this pain vaguely before but actually seeing it everyday at my current job makes me realize there are engineers suffering on both sides of software billing.
Stripe offers simple metering, but in the end, you make the job of calculating metering charges (aggregate all usage and send the value to stripe). It’s becoming harder when you are an API or a cloud company who want to bill a server usage for instance. Calculations are getting heavy, costful and hard to maintain. This is why I do think stripe is not a usage based solution, and also the reason why I decided to create a company trying to solve this pain :)
Usage-based billing is incomplete without its suite of reporting, consolidation and visualization tools. No wonders there is a cottage industry of cloud computing consultants and SaaS solutions focused on helping you to make sense of your AWS/GCP/Azure invoices.
Speaking of usage-based billing, if I intend to charge users a usage-based bill at the end of the month, as far as I have looked Stripe is the only provider that supports this (they call it "metered billing"), is there really no alternative?
Ok for the one-upmanship I raise you billing systems for telecommunications. 1000x the transactions per customer and 1000x the complexity of the business rules.
Eg. An SMS on an off peak time is cheaper. If the user makes 100 or more calls a month a special rate. How do you charge data per plan? If the user roams to another country there's an entirely new set of complex billing rules.
I program billing systems, and even account reconciliation systems for payment processes. It's not a nightmare, it just take a greater understand of accounting. If you want to be an engineer that works on financial systems, take the first year of classes that accounts have to take and pay attention.
This. I'm astounded at the gaps in some people's inability to understand business processes because they have no accounting training. It is not that difficult but it is something that needs to be integrated into your mindset.
You might then realize that accountants have been using event sourcing to solve complex problems for hundreds of years.
Very senior developers will nearly always default to juggling updates to database records for everything. I cringe because it is often the source of serious gaps in domain language between developers and business operations people, leading to unreliable systems that are very difficult to reason about.
You are totally right. But, somehow, most of the engineers working on those topics don't want to work on financial system. They are just part of a company and take this topic "because someone has to take it". They accumulate the tech knowledge, but not necessarily the whole accounting part
Solid job unearthing the numerous billing system challenges (almost triggering PTSD from prior experiences interacting with such systems...). While I haven't fully checked out the details of the solution you offer, I'm excited to see builders embarking on building better solutions :huggingface
I occasionally get PTSD flashbacks from my time maintaining a home brew billing system.
That shit is hard. There are so many ways to fuck it up because you didn’t know what you were doing. The list of unknown unknowns for a billing system are huge unless you are an actual expert in billing, which you aren’t and neither is anybody in your team.
I work on a team collecting fees at a financial company. It is tedious and boring. There is a lot of complexity. I've often asked the business if they had ever thought about a different fee model that would be less complex. They just want to stick the legacy business model into the new tech...
I'm Anh-Tho, one of a co-founders of Lago, thanks for your comment!
Financial and cloud infrastructure companies are the ones with the most complexity.
I've been on the business side, and sometimes you do want to change the pricing but there's so many implications:
- Maybe existing users will churn and/or revenue will decrease as a result
- If you change the billing system, there's a risk of bugs/errors and complains
- If you spend engineering time on this, then you need to deprioritize other projects
So... business teams often end up giving up, and that's a shame because iterating on pricing is a very powerful lever of revenue growth.
> To solve this problem at scale, we adopted a radical sharing approach. We’ve started building an Open-Source Alternative to Stripe Billing (and Chargebee, and all the equivalents).
This is so great. I personally haven't had to deal with this problem, but I've worked at a number of organizations (and heard about many others!) where this sort of business logic had to be implemented. It's just reinventing the wheel. I shudder to think how many companies have implemented a system for managing recurring subscriptions.
We have things like Ruby-on-Rails, Django, Laravel, and many others to take the bite out of building web applications. They keep us from having to reinvent the wheel.
We need similar open source frameworks for common business use-cases - billing, subscriptions, order/purchase management, and so on.
Sometimes it's weird to remember we're in the stone age of technology - all of this is a few decades old at most, and even early predecessors don't go back more than 70 years. Human history goes back tens of thousands of years. There's so much yet to come.
> We need similar open source frameworks for common business use-cases - billing, subscriptions, order/purchase management, and so on.
You mean like an ERP? There's 2 open source ones I can think of:
- Odoo, which is the bigger one, mostly Open-source (CRM / sales / subscriptions / invoices / webshop / inventory / purchase) but some enterprise-only modules (Rental, Field Services), I work with this software daily as an Odoo Developer (Customizing Odoo for customers' needs).
- Erp-Next, which is completely open source as far as I can tell, through my limited testing it seems to be less advanced than Odoo currently.
EDIT: you can even check runbot.odoo.com for some test-environments which are automatically built, where you can test/experiment, login is always admin:admin
This is where solutions like WooCommerce shine. WooCommerce is an amazing piece of software that is many times used for free, but generates thousands of dollars.
Hour 1: 10 KW used for 0.5 hour = 5 KW (10 x 0.5)
Hour 2: 20 KW used for 1 hour = 20 KW (20 x 1)
Hour 3: 0 KW used for 1 hour = 0 KW (0 x 1)
Hour 4: 30 KW used for 0.5 hour = 15 KW (30 x 0.5)
TOTAL = 40 KW used x $10 ⇒ $40
I started my career in Telecoms in a field adjacent to billing systems, and industry lore is that only one in three billing systems project succeeds in delivering anything, and only half of those fail to meet requirements.
Should you compare and contrast off the shelf solutions, test one or two, and slowly expand its use to cover more of your business? By all means, yes.
Should you develop an in-house custom variant after that? Still no. For billing or anything else.
Even a pure tech company like Google should not vertically integrate everything. You quickly end up wasting time making horrid systems instead of generating your main business revenue.
"Oh, but our needs don't align with the way the industry standard third party systems work." That's a symptom of the disease. You won't cure it by enabling the virus to proliferate even further.
People who don't execute I.T. low level changes mistake computers for magic. Yes, computers can do a lot. But even the tiniest feature takes an incredible amount of focus, cost, development time, maintenance time, headaches, and gnashing of teeth to accomplish. The magic is that it looks easy, from the user's point of view.
Let's put it bluntly: We can draw a direct line from third party systems with multiple businesses as customers, with cost savings. It's far more likely to be a complete waste of resources to in-house a new software system.
Maybe you don't have a name for the kind of system you want or need. That's okay. Keep assuming it's probably already out there, and continually search for it. Googling is cheaper than building.
"But we can build it better than them because..." Please, that's the fallacy of exceptionalism. Remember, every company wants to believe they're truly pioneering, talented, and logistically capable of doing all the things. Don't spread yourself so thin. Keep the lights on. Grow your customer base. Everything else is a pipe dream.
"But the third party systems suck."
Yes, they very much suck. And yet the cheapest path to success is pushing them to un-suck their product. Not starting from scratch to create a second, even more immature product.
Frankly, computer systems are already quite flexible. You can often bend an existing one to fit your needs. This happens with Jira, for example. They customize it until it can't function at all anymore. But this sad extension situation is still cheaper and more effective for conducting business than invention.
If you truly believe a new kind of system needs to be built, go and start a new software company. Or pay a software vendor to do it. (Software vendors, just take their money, smile, and nod.) But for Warren Buffet's sake, don't spend a dime on a new line of code if at all possible.
I am so fucking exhausted with the content marketing on Hacker News. It’s not that it has no value. It’s not that there isn’t a place for ‘developer evangelists’. It’s not about this piece. I’m just sick of it.
I come here for a refuge from manipulation- a place where people can share and debate things on their own merits. Content marketing is a not-so-sly way to insert one’s business interests into these types of conversations I enjoy for the purposes of driving metrics, sales, and series A’s.
I hear that taxation and billing and collection are complex. Try to implement WA State destination based sales tax for digital only goods. Where is the relevant tax district? It doesn’t align with zip codes or congressional districts. Municipalities move and reshape the landscape like rivers do. Is a billing address representative of the location of the purchasing party for tax purposes? We took this to a tax judge in 2014, and their final response was ‘do your best’.
I would love to read a blog post with such a war story- about parsing the tax rate soap API into a relational database that can not slow down a checkout page. About how time zones affect rate calculations. Where rate file updates accidentally modified past transactions and how to model data to prevent that. I don’t want to hear something is hard so you should buy my services. Not here. Not on a call to my cell phone. Not ever.
Business leaders need to determine which complexity they want to their developers to tackle- what is the most valuable for profit growth and risk mitigation. If the audience is business leaders, appeal to them directly, and take it somewhere else. If it’s to appeal to software developers, it shouldn’t be attached to ulterior motives. We see through it and we resent it. To the extent that we have influence over purchasing decisions, I’m likely to punish companies who regularly engage in this behavior.
Content marketing is different than creating useful content as a marketing strategy. I’ll raise Digital Ocean as an example. Their sysadmin content is top notch, and I’m happy to give them business for it out of gratitude. That is my decision. Nobody is trying to convince me of how to invest my or my company’s resources.
We get it. Auth is hard. SMS is hard. Billing is hard. None of it is core to most companies core business. If you really want to make your point, show me some code to prove it. Or spend your content budget on something that helps people. Just don’t think you’re going to win me over with a sales piece on HN. You’re not. I’ve already forgotten your name, and if it does come up I’ll have nothing but a negative memory to recall.
In the 00's, I was part of an effort to get the State of Oregon to consider open source software. A 40 million billing system debacle was part of what drove this effort. The details are not important, save to say the ASA paid big to stall the legislation and that worked. It was stalled, and word got out that future bills would be toxic. So, that's over, but a great idea remains!
I am putting it here, because someone, somewhere should do it. Will save us all a lot of money and do a great public good. And it's likely to pay a fellowship of some kind very well over the next couple decades. I'm on a different project and need to finish it, and I'm likely to be paid well on that, so...
Instead of paying the likes of Oracle, or some consulting firm to write and support the billing system, the State should do it. So far, I've seen several bad contracts in my State where the State has a right to use the software, does not own the code, must pay for support, and on it goes. This is quite expensive, and frankly, it didn't work well.
Instead do this:
The State sets up an organization. This org is public, and those who work there are public servants, or some similar arrangement. The key thing is the billing system gets written on open code, uses open data, and the resulting software is owned by the public, through the State essentially. This code gets published on Github under an appropriate open license.
Fund this org, paying respectable developer salaries, and get the system written, tested, deployed, whatever is required. This funding comes from public tax dollars, and I would suggest the lottery as a primary source.
Once the initial system is complete, and let's say it's for water. Once it's complete, the system can be published and the whole thing becomes an export for the State that does it.
Any municipality anywhere can do one of a few basic things:
They can build and deploy the system and handle billing themselves.
They can hire the Org to help them do this, contact for support, whatever.
They can hire private entities to deploy the system.
Changes can come from anywhere, but the organization is best suited to implement them. This can be on a contract basis, funded reasonably in any number of ways.
The organization continues on to develop other public works tools, each funded by public dollars, each displacing some expensive thing or other, and each saving the people quite a bit over the course of say a decade.
Wash, rinse, repeat.
The end product is a nice pile of public works software needed by tons of municipalities most all of whom are paying hard for it now and they pay hard for basically everything, and continue to do that. The public could make these investments once and benefit for a very long time.
Perhaps more than one State wants to be a part of something like this, or whatever State gets going on it first, might want a few organizations, depending on what is to be done and the scope / scale required. Either way, the Organizations provide jobs and consulting / support and basically own the public works they develop and are funded to maintain those software works.
An advantage over contracts and proprietary license is cost, particularly when the likes of Oracle are involved (and no judgement here and no calling out as I'm only saying Oracle as a well known example among many out there and I just feel the public works can be done without Oracle and the cost associated with similar entities). Having the data be open is another advantage in that it can be used in all sorts of ways, and the public records can be made available to the public in simple, effective ways.
Mix in smart API's and the whole thing becomes very high value and an attractive base target for all sorts of public software development having to do with the basic machinery of society.
None of this is particularly sexy, but it does pack a big punch in terms of modeling how a more modern, digital society could employ open code, open data to get the work we need done, done in a cost effective, efficient way.
And that's it basically.
Over time, this first, or core organization will accumulate considerable domain knowledge. And it would become the basis for a lot of other work. Ideally, others started up in various places would all garner the knowledge needed by domain:
Utilities, water, power, internet (where it's actually municipal, and we know that's not popular with the big telecoms...), garbage, etc.
City functions, parking, various permits, anything that has a process that could be automated and or just performed on open code in simple, direct ways that are lean, consistent.
Vehicles: Registration, and all that tends to be tied into these systems. In my state, we've got vehicle registration, drivers licenses / ID Cards that link to facial biometrics (I don't like this, but that is another discussion), and it's also voter information central with a signature, party affiliation, address, and everything needed to manage the voting rolls, and execute Vote By Mail nicely, and in a trustworthy fashion. Oregon has done well on that front. Citizenship records are in this pool of software too.
Health Care: Right now, again in my State the Medicaid system insures people who are diabled, are poor, perhaps wards of the State. This is currently on Oracle, and has been such a mess it's ended up in court. I don't even know the state of it all right now, but it's a lot of money and a lot of grief.
Now, this won't be popular with a lot of us here, myself included, as doing stuff like this will displace some revenue streams companies and people depend on. However, a longer view looks considerably more attractive. The jobs created are needed. Tie this stuff into education, and it's a perfect "farm" training ground for up and coming developers to work on important projects and get good experiences.
A lean and mean digital civics is attractive and necessary for a lot of reasons. The case for doing it on open code and open data is super compelling and compliant with the basic ideas behind public works anyway.
As citizens, we get to experience our taxes actually getting more work done and at a lower cost. And yes, spending will move toward other areas where it's possible to extract revenue, and that's an ongoing problem. Real good can be done, and that's what I'm writing about.
Companies wanting to or needing to interoperate with the State government can do so employing everything from really old school, courier, papers, and the like, through to very new school, all digital, everything operating in ways we know can work well given the incentives are where they need to be and the organization is funded as it needs to be.
The second and third order effects could be very significant! Standards fall out of this kind of thing, and devices, protocols, and all manner of currently diverse and expensive to maintain systems could be made leaner, meaner, have much longer service lives and the knowledge needed will be out in the open and available to the people who need it.
Obviously security needs to be a consideration. And that's no different than what we have going on right now. The big difference is an open society can be built on open code and data and having it be funded in ways that maximize the value to the public while also keeping costs where they need to be, appropriate makes a ton of longer term sense.
In a scenario where this is all closed, the idea of competition is often cited as a reason to do it private, but the realities are corruption and other harsh realities tend toward scenarios where lock in type solutions favor the public getting the lowest value possible for the highest dollar amount possible. We see this over and over and over. And that's why the ASA (American Software Association) opposed the effort in Oregon and another one in Texas with such zeal it was kind of amazing really.
Should we do it in some fashion as I've hinted at here, the opposite becomes true. Initial value for the dollar might not be a whole lot different from whatever contract + proprietary software delivers. Over time, the trend will be toward getting very high value for the dollar.
Regarding Standards... Think center of gravity. Systems like these can have very attractive start costs. And cost of change will vary, and likely be made higher by current players wanting to leverage lock in to preserve revenue streams (and who can blame them?). Fair enough, but as more gets done, that center of gravity will prove compelling, and we all get the benefits working in a similar, more compatible way provide.
When we all require a resource or process, and let's just take power or water for a moment... Running these things at profit puts incentives in the wrong places. Maximizing profit is not the same as maximizing the public use value. And where that is wrong, everyone takes a small hit, and those tend to add right up.
Maximizing public use value can end rent seeking type arrangements that cause more grief than expected. Maintenance is one, tech debt is another of many that come to mind. Where these are out of public view, they tend to get ignored and risks accumulate, until there is an event, and suddenly, those risks play out, and we the public are faced with a large bill, and that's an old story, no need to say more.
I'll end with the belief this seems to be an excellent way to employ open code and data. And one of the big problems we find out there with open code and data is failure to work on "dull" or "uninteresting" problems. How many times do we have to find out that project X is being used by everyone, and nobody really owns making sure it's going to make sense to continue to use it?
Now I will just stop there. That's it. Some municipality somewhere would LOVE to get a State grant to get this started, and some State somewhere really needs this kind of thing enough to bite on the idea, and there are a ton of messes to clean up.
So maybe those should just start getting cleaned up!
Note OP makes a billing product (OSS, granted) so is motivated to characterize the field as complex and hard. In reality billing is just like many other processes humans engage in. For sure it's more complex than computing prime numbers, but hey try writing software to capture medical services processes :) In the end this is exactly what engineers are supposed to do: figure out ways to model/capture/represent complex processes and data.
Or better, buy a billing system instead of inventing one inhouse. Billing is hard to do correctly. For one thing it interconnects with so many different platforms in your company. For another thing, the cost of fucking something up is high--people don't exactly like seeing mistakes in their bill. Hell, think about tax... you wanna deal with that shit?
To do it "correct" you've got to have people who truly understand accounting concepts like double entry bookkeeping, financial reporting, cash vs. accrual, financial regulations, auditors, etc. Unless your company's product is billing, good luck hiring an engineer who knows this shit.
But trust me from experience. Don't build your own billing system. There are several good ones out there that will grow with your organization. Buy one.
Similar experience in some companies in my CV. Billing systems aren't fun to maintain, develop nor debug. They take a non-trivial amount of time when done wrong and if you are building one in-house you'll inevitably run into some mistake.
Buy a billing system, it'll save engineering hours having to deal with it and if you choose right they'll scale with your company.
Not only will you get mistakes, but much worse in my opinion is the opportunity costs involved. You will always be playing catchup to add new (very basic) features that the real packages offer right out of the box.
For example if your homebrew thing doesn't handle price changes, that might take you quite some time to build in functionality. That time spent is time where you couldn't quickly react to market changes, which is money left on the table. Had you bought something that supports a simple use case like "change the price", you'd hit a couple buttons and boom... your product now sells for a different price.
A companies billing system is one of the most important systems in the company. It is literally how money gets into your pocket. If you build it yourself, you will sign up for pissing away a ton of time writing code that literally will let you collect more money from your customers. Had you bought something, you'd just fucking go do the change right away and collect more money from your customers almost instantly.
I totally agree with you, I had to work on the Qonto's billing system (that Raffi is talking in the blog post) and it wasn't fun to maintain. I remember the pain it was to change anything without breaking the whole system, not because the system was bad, but because it's complex and when you build it in-house, you will certainly take some shortcuts that makes it not so flexible!
Could not agree more. Even building your own billing system atop something like stripe is super complicated. Rebill dates, proration, upgrade/downgrade, cancel, updating card on file, refunds, charge backs. These are all things you end up patching as you come across them.
There was a really nice and simple to use billing software that was built on stripe. I used it for a handful of products after I decided to never roll my own again. It is gone now. GoDaddy bought them and shut them down.
I think Stripe checkout is a valid solution now if you don't mind sending your customers away from your site, but I haven't played with it since they rolled it out.
As you say; billing system is tip of an iceberg. There's reconciliation, settlement, payouts, tax, financial reporting and what not. For a small mom-and-pop store, sure build it. But for any half-complex business billing is a rabbit hole.
I'm not sure this comment is exactly in good faith though-- OP does make a billing product but as a result is also uniquely qualified to provide insights into what can make it hard.
This isn't OP trying to claim it's the most difficult problem to solve from an engineering standpoint. Merely why it's hard.
Of course engineers solve problems, hard or not. That's not really being debated here is it?
Hi dboreham, I'm one of the co-founders of Lago here (the OP).
I completely agree with you, billing is not the most complex process of all, medical services may be of higher complexity.
Our post was meant to highlight the discrepancy between the perceived low complexity (lots of teams who have never done it think it's simple) and the reality, with our own experience building a fintech.
I think for some processes (medical services? I'm not an expert in this field to be honest) people might suspect it's going to be challenging.
That was our intention. Basically we think no B2B saas should be building billing themselves, unless they are 300% sure their pricing will always remain subscription based only, and very simple (same amount every month).
If it’s as annoying as translating human processes into code, I can see how they’d want to standardize it so they’d never have to do that particular part again.
Then again, if you make it too customizable, you end up with current single sign on solutions.
I have no vested interest in selling billing products, and my experience is that billing is somewhat unique in the delta between how hard people who have never done it THINK it is, and how hard it actually is. There are many hard things that appear simple, but time and time again I have seen inexperienced engineers be caught off guard by just how much harder billing is than it looks.
This feels like a good moment to mention my most embarrassing bug which was, of course, in a billing system. I had written code to handle the overdue billing of customers' usage on an internet telephony service. The code seemed to work great in QA so we went ahead and pushed it to production.
A couple days later we got an angry call from a customer whose card we'd maxed out. The final stage of the process was to adjust their bill by the amount that they paid, but there was a sign error in that adjustment so instead of lowering their balance by the amount paid, we increased it. As a consequence, for a user with, say, a $50 balance, we kept doubling the amount that we charged them each day.
I think this is a great article and sales pitch for the company, since a lot of the comments have been anecdotes of how its actually even HARDER than the author leads on.
Even if with blockchain and crypto you need to define a logic to bill your customer. This logic is pre-transactional and define how much you need to price the transaction.
Adopt a nothing is fungible, everything is serialised approach and maintain real equivalence classes instead of assuming fungibility as a proxy of replaceability.
Dates are no problem, everything is billed at the end of month, quarter or year.
Upgrades are downgrades are simplified, customer can legally change tariff only on predefined dates/events.
Usage - yes, but exampled in the article do not even scratch the complexity of this
Idempotency is not really needed, the billing process is not continuous, it is a batch job.
Cash collection is out of scope of billing software.
Taxing was clear. Yes, there were multiple billable items with different tax rates, but not too complex. All customers were local.
The nightmare part was:
- customers can have multiple consumption locations, location can have multiple meters and customer can request to split the bill on multiple invoices or combine multiple bills to single invoice as they please
- meters can be replaced at any time
- meter readings are available on dates totally independent from billing cycle. Most of consumption data are mere forecasts.
- when the actual consumption data is available, everything must be recalculated AND compensated (on a correction invoice showing the difference)
- actual consumption data can be wrong and may be corrected at a later date, even multiple times
- consumption points can be added or removed or moved to different customer at any time, but this information is only available after the fact
- the prices can change mid-billing cycle for some customers but the consumption data is not available with that granularity
- customer legal information (company name, address) can change mid-cycle