I once worked on a platform where you could buy ringtones, wallpapers, jokes etc by SMS pre App Store and play store.
Every night the platform would kick off a process to claim money from the telco using some key which was tied to an SMS the user had sent.
That process was responsible for more than $1 million a month, but it was buggy as hell.
Half of the billing attempts would fail, the retry logic was buggy, processes would crash so you would just start them again without proper semantics to ensure you don’t bill people twice.
After a while there were so many retries outstanding that you couldn’t get through them overnight, so we would try in random orders and we would have the process running from 3 days ago whilst try close off today’s bills.
I’m sure people were billed incorrectly and a lot of money was left on the table from users who should have been billed. We never really heard a lot of noise about this either.
So I’m sympathetic to this process and glad that Apple and Google have improved the confidence around mobile commerce.
(Googler, opinions my own. I also do paymentsy things.)
Carrier Billing has definitely gotten better and easier. From the Google Play side of things, there is an API that is used to help with things (no public docs I can find sadly).
Not just back in the day, I have at least a couple of customers of which I know they are losing money because of billing issues.
Why? Because most accounting software is horrible and have minimal to no third party integration options. Many of them don't have an API, so the only option you're left with is manually (!) importing some XML structure every day. The logic is usually very limited so stuff like record mutations are often silently ignored during the import.
The accounting software world is evil. Deliberately engineered vendor lock-in, very limited third party integrations, ridiculous charges for 'custom' implementations, etc. I once build a migration tool for a well known accounting/ERP suite, and my partnership with them was immediately terminated due to 'breach of terms'.
Google themselves stole $75 million from Adwords customers by not coding up a particular kind of refund and ignoring it for decades. It is likely a similar thing is happening with banned account balances.
That’s interesting! I’ve often remarked that it’s a giant competitive advantage for huge companies like Comcast to be able to net however many millions of dollars from billing “mistakes” that never get contested. On the one hand, I’m willing to concede they are mostly probably honestly mistakes and they only make up some tiny percentage of all their transactions, on the other hand, it probably adds up to a decent sized company’s total revenue every year that they basically get just because they’re so giant they can screw up a ton of transactions and plausibly claim it’s not fraud.
There's a lot of banks in various countries whose standard “payment API” is uploading a CSV file with a list of account numbers to an FTPS or SFTP server nightly. And guess what happens if you upload the file twice by accident…
I would be more than happy if they just use FTPS/SFTP. I've had enough nightmare working with few that require proprietary Managed File Transfer software (aka MFT) e.g. HULFT, AS1/AS2/AS3 clients.
Near the end of the article: These kinds of incidents make us wonder the real cost of Code and the more real cost of messing up
Imagine having an app about there ticking away and making pretty decent revenue. Little do you know there's a subtle error in business critical code (e.g. in-app purchases, ads, etc), reducing the revenue by over 50%. This is something I've seen first-hand and have become quite paranoid about. I suspect that it happens a lot. The reality is, no matter how many times one looks at their own code, there is no substitute for continuous testing of typical use cases in a production environment, ensuring that business critical code functions as expected. It's a continious process, because even if the code works properly at the beginng and isn't modified, the underlying OS or API's it interacts with can change and introduce unexpected behaviour.
Add instrumentation. At some point in the code, the data before the bug won't line up with the data after. Watch the metrics and make sure they make sense. Validate the theories your have about what you think users are doing.
I've worked for a company that went bankrupt over some accounting system they'd built and used themselves. It contained a subtle bug that made them think they had more money than they had.
It's really vital to thoroughly test and verify any financial code. Any code, but financial doubly so. And do it again in production. Even small errors can really add up over time.
> In order for Google Play to ensure a consistent purchase experience regardless of where the user purchases your product, you must acknowledge all purchases received through the Google Play Billing Library as soon as possible after granting entitlement to the user.
Can someone who understands this domain better explain why this exists?
A superficial reading of the text triggers thoughts of "fraud prevention", but the code the author uses to fix it is clearly one a fraudulent app can also easily include.
Other thoughts about preventing poor user experiences such as dark patterns to trigger accidental purchases, or bait-and-switch functionality with Fremium apps also don't seem to be prevented by this.
Play Billing 2.0 added the ability to purchase in-app items from outside of the application.
This ACK confirms that the application properly processed the notification and delivered the item to the user. Otherwise, Play assumes that the purchase was lost in the ether sometime before fulfillment (e.g. bug, the app crashed, etc.).
From the post, it sounds like the users receive the purchased item immediately but the app developer needs to confirm that the charge went through properly. Otherwise the funds are returned but the user keeps the purchased item.
Without acknowledging the purchase the money was refunded back to the people who purchased it. And the best thing was the dark theme would be activated even if the user got the money refunded.
I was confused too because it sounded like the developer had to log in and click something to acknowledge the purchases. But it’s the app that needs to acknowledge that it has unlocked its features for the user. Makes sense to me now.
This seems like something that static analysis can correct before the new app is even allowed in the App Store. If you accept a purchase but have no calls the the acknowledgement APIs then bounce with a message. Hell, it could even be a compiler warning. This seems squarely Google’s issue and the refund should be returned.
Static analysis aside, as soon as it happens even once, you'd think that there would be a big red warning on your dashboard saying "X purchases were refunded because they weren't acknowledged". The fact that the author had to figure it out through user emails and negative growth is strange.
Mobile games get frequent refund requests and devs are more than a few steps removed from the actual accounting from app stores. The analytics might throw an event if there’s a hook for a refund request but Apple and Google own the bank in a black box. Basically devs get a monthly statement with refunds subtracted from their cut.
There's a huge difference between the user explicitely requesting a refund, and Google automatically ordering one because the purchase wasn't acknowledged.
I'm not familiar with the API, and I could see both of them looking the same on the dashboard, but Google themselves should be able to distinguish between the two, and one of the two cases is very likely a bug in the app code. I could not see a case where not acknowledging the purchase would be intended.
I was surprised when I read that he moved back from Fluter to Kotlin, down the line I read it's due to APK size.
I'm happy to hear some developers still care about size esp. for mobile, but on the other end, I'm sad because I'm currently using Flutter for building apps and I'm jealous of that 4.5MB difference :D
Kotlin multiplatform should be a great tool pretty soon to share lots of code between iOS and Android withouth losing on app size.
It is mostly business logic you will be able to share this way, but seeing how Compose and Swift UI look extremely similar, the cost of doing the UI twice won't be too bad.
This aside, I am surprised somebody changed framework not once but twice ! That must be a small codebase (or somebody that really loves to experiment).
When people fill up their phone, they look at apps, sort by size, and bigger is more likely to get removed.
For users who haven't enabled automatic updates, they're going to delay updates more often for bigger apps too.
There may be runtime impacts too, but that's outside my area of knowledge. I would guess there are better things to look at to improve responsiveness of an app than simply executable size, but it probably doesn't hurt.
I feel like most phones nowadays don't really care about a 5mb difference considering they have multiple GB's of space. At least on my phone, the space taken by the actual app binaries is negligible compared to media.
One app being 5MB larger is negligible. 50 apps being 5MB larger isn't a huge problem but it is significant. Modern phones have gigabytes of storage but not many gigabytes.
I also didn't see that the difference is just 5MB. Most apps are in the double or even triple digits.
In addition, caring about anything being 3 times larger than necessary stikes me as the right mindset. Individual items may insignificant but they add up. It's certainly better than the electron-ic opposite.
Most people have no concept of how much space things take. When their phone fills up because of too many videos of toddlers bumbling around they assume it's the apps.
Wait a minute. 5 lines of code were missing in the app, causing users to be refunded. So does this mean a malicious user could decompile the app, remove this code, recompile, and get these auto refunds? This seems a way easier mechanism to get free in-app purchases rather than reverse-engineering the rest of the code.
The version without those 5 lines will now be forever out there though. Malicious peers can download the apk and get the premium version for free. This feels like a poorly thought out design on Google's part.
did the same implementation one month ago (and also missed the acknowledgement step :/) and we are 1 month before release, first thing on Monday morning to add and test it. Thanks dude cheers :D
Seems like this shouldn't be the default. As accepting and then refunding a payment is so uncommon, it should require extra code for that case instead.
The issue is that if you have a bug where the feature isn't activated even if they pay, the user would lose money. It's natural that as you activate the feature you should acknowledge the purchase. Otherwise you could have bugs like double charging and such.
It was an unfortunate mistake by the author, but it makes perfect sense why it happened. The person who made the mistake should lose out.
I think it’s for the case where the app crashes after accepting payment but before writing the granted asset token to the user’s profile. No real way around that other than two-phase commit.
Building a successful business requires taking responsibility for all the boring details. Complaining that it is someone else's fault wastes precious time and hurts morale.
I'd have just added "of sales" or "of my sales" after the dollar amount to make it more clear.
I actually looked at this because I thought Google had refunded money to the author and I wanted to see what had actually inspired an act of customer service from a company known for not having it.
Every night the platform would kick off a process to claim money from the telco using some key which was tied to an SMS the user had sent.
That process was responsible for more than $1 million a month, but it was buggy as hell.
Half of the billing attempts would fail, the retry logic was buggy, processes would crash so you would just start them again without proper semantics to ensure you don’t bill people twice.
After a while there were so many retries outstanding that you couldn’t get through them overnight, so we would try in random orders and we would have the process running from 3 days ago whilst try close off today’s bills.
I’m sure people were billed incorrectly and a lot of money was left on the table from users who should have been billed. We never really heard a lot of noise about this either.
So I’m sympathetic to this process and glad that Apple and Google have improved the confidence around mobile commerce.