Author here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky.
Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones.
1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manager had been fired for not being fast enough, and the whole test department was at risk of redundancy. Vice president level executives were asking for updates in daily standups.
Like I say in the post, sometimes, you do what you're told or you find another job. If it hadnt been the height of Covid , I might have chosen option 2.
2. Why didnt we test on a staging / dev server? We did have those, but they didnt work with the K-Pop companies servers. And we didnt have time (or enough people) to make it work. Lots of other engineers decided to just quit. Besides, we never thought K-pop would actually lock phones we didn't own.
3. Some people have asked about it: This type of app is perfectly legal. If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. This is builtin on iPhones, Android need external apps to enforce this.
4. While it may seem stupid in retrospect (LOL, this guy calls himself an engineer, tests in production)-- at that time, it was just one firefight after another, and there was no time to think or plan. I can joke about it now, but at the time, it was one of the worst pressure cooker environments.
Some updates: After we released the 3 products, the whole QA/test department was made redundant. Including my boss, who was fired when he was attending his mother's funeral. Funnily enough, I stayed because I knew details of a legacy product. So far the last 3-4 weeks, I had no boss or anyone to report to.
I think it can happen in every big company, you always have pockets of great culture and absolutely terrible culture once you reach a certain size.
But I agree, it's a definite overgeneralisation to say that all big companies are terrible everywhere.
Having worked for very small companies, and very big companies, there's very much pros and cons to both.
In a small company, culture is far easier to nurture and guide consistently across the entire company.
But then, in a small company, you're also far more likely to meet nepotism based hiring - "my wife's friend who has worked nothing but retail needs a job, and we need an HR manager" .
And also far more likely to work for CEOs who make decisions that adversely impact employees lives dramatically, because they were in a mood, with few, if any, checks and balances.
After all, the new HR manager is a close friend of their wife, their families have been holidaying together for years, and they don't want to jeopardise that friendship by going against the CEO.
I'm not saying that these don't exist in big companies, but generally they have more people in the mix who can say "hmm, let's hold up a second on this".
And to be fair to small companies, they're great for moving into new positions.
> If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it.
No. You don't turn off somebody's phone while they're using it. You have no idea what they could be doing. Someone could be literally dying in front of your locked out consumer who is now unable to call for help. This is vital infrastructure and you're denying them access. If somebody dies because of this, it's on your conscience.
Would you randomly disable somebody's car's engine because they didn't pay? No. You call the police and get them to repossess the car. The police will do it properly without risking collateral damage.
1) I highly doubt that the app disables emergency calling.
2) Some (rental, probably also lease) vehicles actually do have disablers. They prevent the vehicle from being started, not stop them in the middle of the road.
3) The police don't repo a thing. That's a civil matter and generally outsourced to the lowest bidder with a tow truck.
By the way, if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it. Just like if someone died because the car got repo'd and they couldn't drive to a hospital.
1) You don't know that. And there may be emergency situations where you're on the phone with someone other than from the emergency number.
2) How rental cars do it doesn't seem to have anything to do with this.
>By the way, if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it. Just like if someone died because the car got repo'd and they couldn't drive to a hospital.
This is some fucked up reasoning. People run into financial difficulty for various reasons, sometimes out of their control. But even for the serial deadbeats who don't care about financial responsibility saying "yeah, it's fine that somebody or their loved one died because it's their fault"? I can't imagine thinking like that.
> Would you randomly disable somebody's car's engine because they didn't pay? No.
Perhaps you meant to reply to the person I was replying to, rather than me, with your "how rental cars do it doesn't seem to have anything to do with it"? Because the person I was replying to said that repossessors wouldn't disable a car, which is not true.
> This is some fucked up reasoning. People run into financial difficulty for various reasons, sometimes out of their control. But even for the serial deadbeats who don't care about financial responsibility saying "yeah, it's fine that somebody or their loved one died because it's their fault"? I can't imagine thinking like that.
Saying "yeah, someone else SHOULD pay for their non-necessity"? I can't imagine thinking like that.
Just because someone runs into financial difficulty doesn't mean they can just stop paying for things. Nor does it mean that anyone else has an obligation to provide them with something for free. Particularly when that something is a luxury. Heck, if they're so short on money they could've sold their expensive Samsung phone and replaced it with a cheap one.
And again, I highly doubt that any of this is even a possibility. And you can say I "don't know" that it doesn't disable emergency calling (never said I did), but the fact of the matter is, it's virtually impossible to disable emergency calling on any phone, thanks to both regulation and technical standards.
>Saying "yeah, someone else SHOULD pay for their non-necessity"? I can't imagine thinking like that.
>Just because someone runs into financial difficulty doesn't mean they can just stop paying for things. Nor does it mean that anyone else has an obligation to provide them with something for free. Particularly when that something is a luxury. Heck, if they're so short on money they could've sold their expensive Samsung phone and replaced it with a cheap one.
That wasn't remotely what I was saying. I wasn't advocating for anyone to get non-necessities for free nor that difficulties automatically absolves people of responsibility.
I was saying that to dismiss any consideration of life and death situations because any deaths would be on the people not paying is fucked up.
Is not donating malaria nets to poor africans more or less defensible than choosing to sell them phones which deactivate when unpaid?
You seem to be suggesting that it would be fucked up to not donate these phones to the poor people who aren't paying their bills. Does that mean that it's also fucked up to not donate them other things?
Nobody's saying they have to give out phones for free. We're saying that somehow those people ended up with the phones in their possession and the company's right to get paid DOES NOT mean it can lock out emergency functions on the phones. They can go ahead and cut off the normal phone service but if they lock out the dialer app and even one person fails to call an ambulance or police or firemen or whatever because of that lock that's one hundred percent on them. This goes double, triple if the lockout happened by mistake to innocent people.
Honestly, it's bad enough that these companies think it's even acceptable to remotely control computers. Locking out emergency services is just on a whole new level. Didn't get paid? Seek redress in the justice system like everyone else.
Okay. It's right there on the title: "my script killed 10k phones". Do you have any reason to believe "killed" means anything lesser than "rendered completely useless, bricked"? Also, there's the fact the software "would lock the low-level features that allowed you to make calls". There is no reason to believe calls to emergency services were exempted.
I don't really want to comment on the rest of the article. Actually, as one of the "poor sods from places like Peru and Chile" I do have quite a few things to say about the author's gross negligence and his endless rationalizations. I just don't want to get permabanned for saying them. Suffice it to say that negligence of this magnitude is actually a crime where I live.
> By the way, if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it. Just like if someone died because the car got repo'd and they couldn't drive to a hospital.
Sure. What they were thinking trying to buy something without having money, right? And even worse, what they were thinking being poor. They should have bought a very shit phone, closer to what they can afford. And if they lost their job and then couldn't pay for the phone, it's their own fault. /s
If everyone saw things like you, there would probably not be nearly as many poor people with phones.
It is these very consequences (repo, deactivation) that make it possible for these phones to be offered at affordable prices.
Sadly we don't live in a utopia, especially the poor people of this world don't. I'm sure they'd rather have the phone that stops working and possibly indirectly kills their family when unpaid than no phone at all.
1. Lots of poor people have phones, sometimes very expensive ones (as these are sought as a symbol status)
2. There is no such thing as repossession. And living abroad makes me believe that it is an phenomenon in United States only. It is illegal in many parts of the world.
Not only there is no repossession but almost everything is sold in installments. You can pay in installments even in supermarkets.
In that case what makes the system works is the credit system. When almost everything you buy you must use installments then losing your credit is a terrible situation and the only solution is to pay (or try to).
In the beginning of my career I worked in a small local ISP and usually the middle/high class were a bigger part of the people with overdue payments, while people living in worse neighbourhoods were almost always paying in time. They can't afford to not pay.
Sometimes we live in a bubble that makes we think the way we live and the way things are are the only way things should be. We must be careful to not extend our vision of things to everyone and everything, the world is much bigger than our backyard.
> I highly doubt that the app disables emergency calling.
Well, I highly doubt that it doesn't. Given how little attention the company paid to these important details, who's to say they didn't simply brick the user's phone?
> They prevent the vehicle from being started, not stop them in the middle of the road.
Like I said, nobody will randomly disable somebody's car engine. These people at least have enough foresight to wait until it's safe.
> The police don't repo a thing.
They do where I live. Bank loans money with car as collateral, consumer defaults, bank seeks justice, judge authorizes police to repossess the car so the bank can sell it and minimize losses. Police can enter private property and even use force if necessary.
What is even the point of police if they aren't gonna do things like this? Outsourced to tow trucks? Really? When some random citizen reposseses property, it's robbery.
> if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it
No. The fact that person didn't pay for the phone doesn't matter matter at all. The company can always negotiate new terms or seek justice later.
What matters is the fact that in an emergency the only thing standing between the consumer and a call for help was the phone company. Worse, they got in the way on purpose, as if it was their job to punish the consumer. There is no defending this.
> Just like if someone died because the car got repo'd and they couldn't drive to a hospital.
Not really. If you don't have a car, you can still call emergency services. At least you would have been able to if some stupid company had not locked your phone.
> Would you randomly disable somebody's car's engine because they didn't pay?
I believe that's exactly what some systems for subprime car loans do (they don't shut it down while running, but will prevent it from starting - if they're somewhat clever then only if the car has been off for a certain amount of time).
You know there is literally a 0% chance of the company that owns the phone getting it back after having it stolen from them. Having a system like this is the only reliable way of making sure people in low trust countries will pay what they bought.
So what? Don't loan out phones then. If you do, then don't lock out emergency functions. Incorporate the risk of not getting paid into the product's price or something.
In my country, if someone's life is in danger and someone else sees that, they are legally obligated to help. Failing to do that is criminal negligence. When those people are in front of a judge being asked why they didn't help, what will they answer? "Oh, I tried but the phone company locked me out." Will the judge care about the fact that the company locked them out because they owe them money? No, the judge is going to make it even more expensive for them at the very least and that's honestly a pretty generous outcome.
Failing to call for help because you don't have a phone? That's tough, a difficulty of life. Having one but being unable to because some idiot company locked you out? Yeah, that's extremely fucked up. Emergency services are rights and no company on earth gets to deny people those rights. Nobody really cares that the user didn't pay for the phone. What matters is that the phone was in the user's possession and could have saved a life had the company not locked the user out. Nobody deserves to die because of some company's entitlement.
Just replying here so you can see it, but I would caution against continuing to share this story as it's written. If I was a hiring manager and I saw this, I wouldn't hire you, regardless of how well you did during interviews.
These are my options, some shared by others in the threads here, some opposite.
> It would lock the low-level features that allowed you to make calls, use Wifi, or even post pictures on Instagram/Facebook (the horror!) until you paid up.
...
> Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him.
Both of these quotes seems like you don't quite understand the impact on your end users, or just don't care. Neither are good options as you grow your career.
> Our own internal tool worked well (I would like to say it was all due to me, which would be an arrogant thing to say, but also 100% true, so I will say it: It was all or mostly due to me).
Your internal tool, which you are taking complete credit for, is what didn't have protections in place that ended up harming users. Also, it comes across as bragging or something. People that are too obsessed about credit usually are not ones I'd like to work with. Team efforts are hard to define and split up and people that over index on that usually are just bad team members.
> But one mobile phone manufacturer (that I will call the K-Pop Phone Manufacturing Company), based in Korea (yes, that one)
Say it or don't. These games where you can act all cheeky about how you didn't 'actually' say it is just juvenile IMHO.
> Because of time pressures, there was no time (or political will) to check the script was well written. As soon as I banged it out, it was live. And I mean literally, 10 seconds after I pressed save, the script was running on live production servers.
This is beyond just the org's failure. As engineers, we're responsible for what happens on our watch. We hold the line and try to enforce engineering practices, not shrug and yolo and then blame the org.
> I could have tested it better, but that would have meant working late into the night. No thanks. I had already worked a few late nights/weekends and I was done.
This comes across as again, you don't actually care at all. Not a hiring quality IMHO
> My colleague was helping me, so we got it all done in about an hour. By 18:30, we had unlocked all phones.
Oh no! You worked an entire hour between 5:30 pm and 6:30 pm! The horror! Most people work much much longer hours, later into the evening or night when things fail. The difference is they don't complain about it and care.
> A project manager confirmed it, and I took a breath of relief and quickly logged off before more fuckups could be found.
Great! You fixed it, but rather then trying to verify you fixed everything, you logged off at 6:30 and screw anyone else effected. Not a great look.
> The Lessons we learnt (or more likely, didn't learn)
If you didn't learn the lessons, why are you writing about it? Just as a war story? Or do you think this comes across as a positive for your career?
> The lesson here is: Do some f'king checks before you lock someone's phone, but I don't know if that lesson applied to us or K-pop (most likely, both of us).
You know. You're in control of what you can do, so you do what you can to prevent issues from happening under your watch.
> This is not just good QA, but good security as well. But bah! None of that silliness for us, thanks.
Again, you come across as uncaring. Why are you working in this industry?
> We assumed that because our own software checked for this sort of thing, that the K-pop phone's software would do as well.
Great. Your software failed. Don't blame downstream if you can't even get your first layer correct.
> Everyone pretended it was an honest mistake.
It was an honest mistake. The difference is if you learn and grow, or if you just shrug and make the mistakes over and over again.
Honestly, this is just my two cents. Take it all with grains of salt, and ignore as you wish. I don't intend to be personally attacking you or any of the sort. Just expressing where my mind is reading it.
If I was you, I'd take this down and seriously consider if engineering is the right career path for you. It doesn't seem like you actually enjoy it or care about it and just working for the easy money doesn't lead to a fulfilled life at the end. And living a fulfilled, happy life, is the most important thing after all.
> Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him.
There are countless situations where suddenly not having a working phone means a serious problem. The article presents an extremely negligent look at the issue.
There are indeed large ranges of highly valid reasons why having phone/ Internet connectivity is highly critical.
But sure, "post duck photos". I understand it was perhaps meant as a paraphrase, yet it is indicative of an almost callous level of arrogance at the author's company.
It doesn't seem unusual to me. You write a test script that generates random numbers and saves them to foo.txt, then reads them back from foo.txt for the purpose of running one kind of test. Then you wrap that in a loop for every kind of test. You end up testing everything, but foo.txt only has the numbers from the last test.
To be clear, I'm not saying it's not a bad practice to do that. But the context is that it was a one-off job to do the testing, not a proper test framework, for which this kind of half-assed approach is reasonable.
True, I think I have something like that in multiple places for tests that are used to validate prod code. But our code won't lock people's devices down with no recourse.
I think what made me scratch my head is that they didn't consider that generating random phone numbers might lead to collisions. I've been hesitant to publish fake datasets that have randomly generated and validly formatted SIN/SSN numbers in it for that exact reason.
Even if the risk is low, I'd still think keeping indefinite logs of that would be a good idea.
If you haven't killed some production environment in your career, can you really call yourself a programmer? I kid of course, but I've been there, done that. It's lessons learned the hard way and I became a better developer because of it.
I once ran a query on a live production web site thinking I was doing a SELECT but somehow ran an UPDATE which set all the millions of users' gender to 'M', regardless of their selection.
Restoring from backups would have been a nightmare. And telling my supervisor would have been a nightmare.
So I used everyone's selected title (Mr, Ms, Mrs etc) to reset their gender setting, and then put the keyboard down and went to bed.
Because zero thought went into the account signup form. It just asked for whatever the coder who wrote it thought it needed. Shit, it might even have been me who wrote the signup. I don't remember. This was back in 1999. Now I try to ask for the absolute bare minimum of information to onboard someone. No information at all if I can! Times have changed. Damn, I don't even identify as a binary gender myself anymore and hate forms that ask me.
Having worked in a healthcare setting and having had a patient come through who had undergone gender reassignment surgery, it turns out to be easier get the surgery than to get their gender and name changed in the healthcare database.
I said I’d help them get it fixed, but I couldn’t get this done.
This case study had a lot of valuable insights cooked into it, so I'm glad this developer blogged about it.
The myth of the developer who knows everything and never makes huge mistakes is a lie that needs to perish. I'd much rather work with someone who is upfront about their mistakes, the gaps in their knowledge, and can speak directly on serious issues of overwork, managerial incompetence, bad clients, and systemic fragility.
While I agree with the overall point of your reply, it would also be nice if we stopped normalizing some of these fuck ups.
Some errors actually happen due to professional negligence, and we should hold ourselves to better standards than "oopsie, updated a whole column in a table".
Oh yes. I've accidentally run a SQL UPDATE command without a WHERE clause and blew away everyone's password. Our process to restore a backup was very manual and time consuming back then, so it was fun sitting around and waiting for that to happen while everybody was locked out of the application.
That's a fair question. Loss of a couple hundred thousand US dollars during an 8 hour window where a poorly performing SQL query during the payment pipeline caused purchases to hang. Indexes are great things... if you use them.
“the main product was an app...that would lock the phone if it was stolen, or if the customer stopped paying. The app was built as part of the Android OS, so you couldn't uninstall it. It would lock the low-level features that allowed you to make calls, use Wifi...until you paid up.
All good.”
No, not all good. There’s the problem right there. If I stop paying you can stop providing me the service I’m paying for. Using WiFi has nothing to do with you. Breaking my property this way, without a court judgement allowing you to do it, is not only unethical, but should be a criminal offense, and is at least a civil tort.
The potential liability here is literally unlimited. I believe responsible cell providers allow you to call 911, or the equivalent in your location, even if your account is delinquent. But you can’t do that if your phone is locked in this way. Did anyone die because of this product and this guy’s recklessness?
Why do you assume the users own the phones? There are plenty of business models where the company owns the phones and the user gets to use it by paying a monthly fee. One of the services being provided may well be the use of the phone itself.
Even if/when they don't fully "own" the phones, users ought to always be able to access their data (at least for backup) and call emergency services (which is a legal requirement in many countries).
More generally and regardless of time pressure, I can't believe someone can "play" with such test a script (used with a production tool) without even thinking it might fuck up and hit real phone numbers... (and if the developer was aware that his approach might fuck up, then he should make his boss aware that he had no time to fully test the feature rather than play russian roulette. Time pressure is not an excuse for everything...)
Because (pardon my language) fuck that business model, that's why. It's reprehensible on a number of different levels, and specific aspects of that have been made illegal, but companies keep trying to bring back the exploitation.
Just because there's a business model doesn't mean that we have to support it, or help it be implemented. We can shame our fellow developers choosing to enable scams and write malware/computer viruses.
Lets say I start a business and the business model is going to be "every time tcskeptic posts a comment to HN, they give me $1". Sounds pretty great to me! As part of this contract, no objections to this business model will be allowed. There's the small detail that you didn't agree to this contract, but hopefully it illustrates that just because I have a business model doesn't mean anybody has to accept it.
Put frankly, you're out of touch with how south america works. Things you would never expect can be paid in installments here. People just don't have heaps of disposable cash.
There's litterally lines at the atm 2 times a month, pay day. The rest of the time people are day to daying without cash in their pocket.
It’s more like either I spend $500 on this device and own it or I spend only $100 but have to deal with ads. Yeah I hate ads but I can use that extra $400 to fix my leaking roof.
That would be something to discuss with the user and the company that then might own the phone. In this case though, someone completely unrelated to the contract locked the phone. Someone in another country. Someone working for some completely unrelated telco company. Someone who just randomly locked some phones by banging out a script to generate phone numbers, without checking if the phones were owned by the user, their telco, or my grandmother.
Huh? So you're telling me that I can walk into the phone shop, get them to give me a phone with financing, and then not pay it, and... still own it? Sweet deal!
Correct. Which is why they instead simply disable the device (and this has been done for quite a long time, by the way, with banned IMEI lists, back before phones were useful for anything without a cellular connection).
Normally they'd come after you with debt collection agencies / court cases rather than asking for the handset back. They want their £40/mo for 24 months, not a used handset really.
"You may get Equipment from Us directly when You take out a Price
Plan, whether for free or for an upfront cost. Unless We tell You
otherwise, or You get Equipment under a Finance Agreement, We will
own any Equipment provided to You by Us for the first six months of
the Minimum Term."
> We will own any Equipment provided to You by Us for the first six months of the Minimum Term.
But yet they all make pains to separate airtime, and devices in the contract for example clauses that say you can't cancel the airtime contract if the phone proves faulty
The author works in the UK and there's one company listed on his LinkedIn that provides the services he described, which is situated in Cambridge. The description of the product seems to indicate that the software is written to allow carriers to lock out people who stopped paying, mostly targeting people who'd normally have trouble getting credit for their phones.
Unless the company specialized in sales outside the UK and likely also outside the EU, I think this company is probably breaking the law, as are the carriers using their services. I doubt the people who buy phones on extended credit will have the cash laying around to bring up a lawsuit, though.
The phone numbers generated for testing were in South America, but the app can block anywhere in the world from any country in the world, including UK. That stuff blocked random phones on the globe, if that is not clear enough.
English Common Law is prevalent, but that is the basis of the law and certainly not the specific codes and minor changes that have been made since 1950
> If I stop paying you can stop providing me the service I’m paying for
Indeed. In many cases, that service is the phone itself, given that it is leased or financed through the carrier. And in those cases, based on the contract which the customer signed with said carrier, ownership of the phone generally reverts to the carrier if you don't pay up, like with any other secured loan.
The author blames this on management pressure and deadline acceleration but it seems to me that they made an incredibly obvious and negligent mistake and is trying to pass the buck.
Exactly! There seemed to be a severe lack of any production control processes at this place. That's the fault of management and senior developers/architects. Contractors should never be given the keys to the kingdom like that. And if they are, then there should be some checks and balances to make sure they don't do something foolish like this. But I've worked at places like this where we coded "close to the wire" so to speak and it's just get the job done and who cares about quality. Management should absolutely get the blame when a screw-up happens.
I am not sure how a non-technical director is supposed to hire a software engineer if software engineers themselves fail to hire good software engineers.
They mentioned other people being fired for "not being fast enough" - I'd say that puts the blame squarely on management and not them. People will do all sorts of silly things in a culture of fear and unmovable deadlines like that.
The deadline for generating a bunch of random phone numbers to block? The author would probably save some time by hard-coding some truly non-existent phone numbers instead of using a random number generator.
Author says in the article they had a 3 day deadline to finish the product and needed to test thousands of cases while already having pulled late nights to keep up with schedule.
“To the tens of thousands of people whose phones were suddenly and inexplicably bricked: my bad. You see, it was almost the weekend and although I had 3 days to responsibly approach this problem, I had already pulled several late nights, I’m sure you understand”
That's not the point. It's just not sensible to blame this one developer for a horrible environment in which this was the only option aside from getting fired, which is quite onerous a responsibility to put on a single person. This falls squarely on management.
>in which this was the only option aside from getting fired
I don’t believe an ill conceived script was his only option. And the rub for most people in these comments is that the code he wrote should have never made it to his fingers—he *knew* it was going to be rushed into prod, but he didn’t stop and consider that randomly generating numbers in a phone number-like schema to the scale of tens of thousands would actually semi-regularly produce a real phone number?! Seriously?
Anybody that has been dealing with deployments in critical systems has been in similar situations, specially being a vendor. When you are there it all depends on your risk appetite.
What on Earth? This company just locks random people's phones as part of their testing, and nobody realised that was a problem until they did it to thousands at once?
I think that's the real issue here. The Korean company (probably Samsung or LG based on the implications here) didn't validate that this company had permission to lock these phones.
Things like these can happen by mistake (or incompetence, in this instance) and the unnamed company should have never let this happen.
I believe everything you need to know about who would be responsible for this kind of "mistake" is right here from the article:
> Our own internal tool worked well (I would like to say it was all due to me, which would be an arrogant thing to say, but also 100% true, so I will say it: It was all or mostly due to me)
This is a person who is incredibly overconfident in their limited skills and hasn't yet developed any sort of self-awareness or self-reflection. The fact that the rest of the article blames everyone else for his mistakes further proves the point.
> I could have tested it better, but that would have meant working late into the night. No thanks. I had already worked a few late nights/weekends and I was done.
People in the industry tend to laugh at having to work late as some sort of standard practice in the industry. But if the time allotted to program it is not enough time to program it in a properly tested manner, that's a management problem, even if the programmer could have worked overtime and gotten it tested better. It's a bad practice for managers to schedule things such that they can only be completed with overtime, and managers bear a huge portion of the responsibility for any problems that result from that practice.
This feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.
Not defending him, but he does state that unlike the other mobile providers he's worked with, this K-Pop company's backend system would lock _any_ phone number, not just the ones they're responsible for. Perhaps he didn't know this. I bet normally he'd generate random numbers not associated with the provider, check they show up as "locked" in the database, under the assumption that the backend actually doing the locking signal would reject the numbers due to them being under control of another operator.
It's similar to our networking lab where we generate bogus traffic using random src/dst IP addresses under the assumption that our null routes and firewall will block them from the real Internet. In this case, the provider just decided no firewall was needed. We can't use "fake" IP addresses (yes there are IP ranges for internal use but that doesn't work for certain testing scenarios.)
The fact that he was running production code and not writing the result to any kind of log file, which would only take a couple more lines of code, one more dependency (if he wanted to skip some time rolling his own), and a couple of megs of storage, is so absurdly irresponsible, I’m surprised he wrote this post up at all—most would be too embarrassed.
Too me this is another example showing how much behind "software engineering" is behind most otherl engineering disciplines. Processes in other disciplines are build very much into the practice that these things don't happen because people die if a plane crashes or a bridge collapses. Also that somehow software gets away with absolving itself of any liability (i.e. the usual "this software is provided as is...") is quite remarkable.
You're just not looking in the right places, and your comment is a bit disingenuous. You're cherry-picking examples that don't represent an entire industry.
NASA for example has an _extremely_ tight tolerance on software specifications, testing, and proofs [1]. Medical devices and other core infrastructure also has very robust development cycles to ensure bugs don't bring down the power grid or overcharge someones heart in a pacemaker.
Meanwhile you can look at the auto industry and find example after example of shoddy engineering or manufacturing processes putting hundreds of thousands of drivers and families at risk. [2] [3] [4]
People die all the time because things for buildings are changed "in production". It's common for the builder (or supplier... the people in charge of taking the engineers plans and actually building the thing) to swap out parts at their own discretion. And there's tons of stories of lots of people dying because of it.
There's also stories of architects designing buildings using "new and interesting techniques" with little to no testing. There's a story of one building that was lighting things across the street on fire because it was basically a big parabolic mirror.
And.. well.. the Boeing 737 Max.
So sure, engineers and related jobs are _better_ about it, but they're certainly not perfect. The still do stupid things. It's just less likely because the fields have been around for a long time and there's been time to learn the right ways to do things... and to learn the things that get people killed.
As the expression goes, regulations are written in blood... because new regulations come about when people die because of a gap.
It’s like trying to breathe underwater. Just because a person has the capability and know how to breathe, they can’t breathe if not given the proper environment to do so. Yelling at them “why didn’t you take a breathe?!?!” As you watched them drown underwater is just ridiculous and naive.
But every time proper engineering doesn’t happen, we see a subset of cries like this, like it’s some moral failing in the engineer. There are surely some chronically stupid engineers out there, but 99% of the time it’s a culture that has subverted the doers below the discussers and the deciders that leads to this shitshow of “how did this happen??”
I don't understand how people can defend this developer in the first place.
I could understand if they were about 16 years old and actually abused by their employer, but generating random numbers to test the locking of phones? Basic maths should tell you that the probability of matching some real phone number is actually quite high.
Moreover, what exactly were they testing? i.e. if they were testing nonexistent numbers, then how would they even know the phones were blocked in the end?
> Basic maths should tell you that the probability of matching some real phone number is actually quite high.
Yeah well, the thing about pressure (of the sort the author has describing) is that you make mistakes - the kind where some times 3+3 = 9.
Source: I queued jobs to nuke the whole production database (relax, we had automated backups+verification so we recovered fast enough; and then added some gating so make this more difficult in the future)
Why would he write and _publish_ this?!? Is this some kind of humiliation fetish? These are things that should maybe be admitted in a confessional, or therapy.
Being somewhat generous, I don't think it's because they didn't understand the probabilities, it's because they didn't think about the impact of what they were doing at all, beyond the problem they needed to solve.
I.e., the only thing they were considering was whether the number entered at the start of the test would show up in the right places at the other end of their system. They simply didn't consider that an actual phone would be shut off along the way.
I also get the feeling that the author didn't learn much from this experience. The lessons learnt here are "should do better testing" but even that didn't stick based on the paragraphs after that.
The complete disregard for the victims of this company's incompetence ("the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram") makes this a very different kind of article from the usual "I fucked up production the other day" posts that you see here now and then.
My conclusion after reading this is that the author didn't learn anything, the company didn't get punished enough to re-evaluate their testing and the Korean company's systems are still vulnerable to unauthorized locking.
The real lessons I've learned from reading this are to avoid either LG or Samsung and any provider using Trustonic. And not to hire the author, I suppose.
This is a horror story, not because of the author (sorry no sympathy for you mate), but because of the dystopian nightmare that we've turned commercial transactions into.
People presumably bought a phone, and because of some nonsense "business" plan, there is a recurring charge they are still liable for, and the manufacturer can arbitrarily lock and disable the phone?
I know there are a bunch of "explainers" here on HN, happy to jump in and say that the phones were subsidised, consumers wouldn't get access to the phones if not for this scheme, and so on. But please stop that and take a step back. We've perverted the traditional expectation of any commercial transaction to own the item to instead be a permanent debt for regular consumers.
You will own nothing, and you will be happy -- indeed.
I've noticed more as time has gone on people are working on critical software without the skills or organizational structures required to actually do so safely. What I'm wondering is if this is a change that's happened more broadly or is just something that I didn't notice before? Overall the average years of experience in the industry has dropped in the last decade due to the influx of new developers but what's the deal with inexperienced people working on critical systems without oversight from people with more experience?
While I feel the failures from the developer here are pretty bad I feel the organizational factors that allows this situation to happen are far worse.
It wasn't so clear but I was mostly thinking of the management when I wrote that comment. In this particular case I think people might get caught up on what an individual did but I think the management failing here is the far bigger deal.
The conclusions made in this blog seem wrong, it's perfectly fine to write these kinds of tests if you have to as part of a rush job, but WHY DO IT ON PROD? You know 100% that you are making calls to the 3rd partys' prod, assuming "random" numbers aren't real is incredibly negligent.
As others have pointed out, I wouldn't even have created this article, but to each their own.
I'm assuming this client company was Samsung, due to scale and the author calling it the "K-Pop" company. They are known for using idols in their marketing. Still a little cringe to call it that.
My take away from this article:
1. This act was extreme, bordering on criminal, negligence. Some peoples' livelihoods depend on access to their phone. I assume most people using this service are of a lower income. I wonder how many jobs may have been lost due to their not having access to their mobile for a day.
2. Samsung needs to hire better contractors -- but this is what happens when companies go with the lowest bidder, I suppose. I've seen it time and time again.
The main lesson here is do not, if you can possibly avoid it, accept a job at a company with a level of engineering quality as low as described in the article. You probably won't get fired or taken to court; the main problem is that you will have no-one good to learn from and you will lose terribly in career satisfaction and future earnings.
If you're new to the software engineering job market, try to hold out for a company with a real team, and some colleagues that you look up to technically. It is hard, the jobs that are easier to get will be tiny companies where if you even have a colleague in engineering, they will be clueless.
"I validated the data to the best of my ability, but I had to rely on otrahuevada's experience in choosing the data. Unfortunately he really screwed up."
If your manager is the type to throw people under the bus, there's always a way.
I mean yeah when there's a will there's a way, but keeping your manager and reports informed about your work and its progress is kind of the only way to cover your ass available to you as a non-manager.
Importance of testing has been told multiple times,even though we all ignore it largely.
I vividly remember an incident ( this was in the early 2000's )
we had a couple of Linux machines both remote and local.
The testing and development happens in Local and the code is pushed in a short time over dialup connection.
One of my team member had a instance of remote open.
He was deleting some cache files in his local , he by mistake typed in rm -rf / instead of rm -rf * in the remote machine ,assuming it was his local development machine.
He didn't realize the mistake and finding it was taking much longer than expected he cancelled the command. went home happily. He was using a common login. Though he informed his lead , the tech lead also didn't realize the seriousness of the command that moment.
What actually happened was every file which had 777 permission ,got deleted till he pressed Crtl-C.
Now some of the files in production server was gone and some were still intact.
This led to lots of confusion , the person who issued the command didn't realize his mistake , till the server admin who was about to loose his job for faulty permissions some how convinced the management that someone had issued a wrong command , and that he was somehow not at fault.
Our Boss was super irritated when he found out what happened. The person who typed the wrong command , along with his tech lead left the job after a few months on their own.
It taught very valuable lessons, Especially not to type rm -rf without checking multiple times.
I still remember this incident every time I delete something.
> It taught very valuable lessons, Especially not to type rm -rf without checking multiple times.
"Measure twice, cut once."
One of the many uses I have for a mindmap in my daily use (Freemind 0.9.0!), is as a place to write out sysadmin/sql commands in a structured way before executing them.
If I can test it on dev before production - great.
If I have to run it directly on production, following this method (religiously!) has saved me from any catastrophic (read unrecoverable) issues.
When you are under pressure it can be easy to say 'I don't have time to write out the commands'.
When touching production servers, you need to spend the extra minute or so to write it out.
I have mentioned freemind a lot in my posts, but I guess I have never really gone into how I have used it over a really long time.
First started using it around 2012 and have been building my local knowledgebase/library ever since.
With full text indexing on I can search a local directory for when I have (successfully!) used something like mysqldump and the context it was used it.
Have found this is significantly faster and more reliable then just trusting google/stackoverflow - especially for edgecases I have solved in the past.
I use freemind 0.9.0 (self-compiled) as I have found it is the most reliable and never crashes version.
Currently running with jdk1.8.0 with an override script on OSX Catalina.
How I use it:
- program.mm
e.g. mysql.mm
Evolved into program-feature.mm
e.g. mysql-mysqldump.mm
Evolved into program/program-feature.mm
e.g. mysql/mysql-mysqldump.mm
- as a daily worktracker (week ending on the Sunday)
e.g. weekend-20121024.mm
Each day gets it's own tree/node:
weekend-20121024
Mon 18/10
JIRA-123 Fix database blah.id issue
Each node can then be written out, colour-coded (red for copy-paste bash commands, light blue for sql), and finally copy-pasted in a chunk in order (windows users be careful of selection order).
Just did a wc -l on my 'program' mindmaps since 2012:
2604
I think I originally found it on a /. story sometime in 2012 and tried it out as more organised local txt file documentation system.
And I loved the fact that I could write out the 'plan', test it, and then copy/paste into a terminal.
I really started it as I was compiling from source a lot (still do) in those days, and there were always little quirks that had to be worked out on different platforms.
For example the mysql/mysql-compiling.mm mindmap was created to provide copy/paste installations for Percona-mysql on Linux and OSX.
The weekly/daily mindmaps are my primary working space.
I wasn't running Jira at the time, and Freemind is faster anyway.
When writing something new in a daily node, I can then copy it into the appropriate program/program-function.mm
Then I started experimenting with colours and icons and came up with my own system:
- green (F3) for search questions
- red (F4) for bash cmds
- blue (F7) for sql cmds
This is pretty awesome. Thanks for answering my questions, it's been insightful. I've been messing around with Freemind for the past hour and I'm liking this. This has definitely inspired me to experiment with it further.
Back in time I was working on an online trading card game, the integration test was running in the live server. (Don't ask me why. All I can say is that the server had to be hastily built for some reason and it was known to be suboptimal in several aspects.) In order to avoid inference the test created tons of placeholder accounts and corresponding environments and it actually ran mostly well given the circumstances.
One day I received a report that several accounts were simply disappearing and had to be re-registered every few days or so. I had to go through lots of recorded logs to determine what was happening: one of these accounts was named "Card" and some test was using it as a placeholder account, so it got removed when the test finished. How apt it is, that someone would use "Card" or similarly rare handles in a TCG. The testing infrastructure was updated to ensure that placeholder accounts can't coincide with actual accounts and affected users were compensated.
> Our company had been bought by an investment firm, and they wanted their pound of flesh. All projects deadlines were moved up, and at one time we were testing 3 products in parallel, all with different requirements.
What are you supposed to do in such a situation? Jump ship? Or attempt to provide pushback against unreasonable practices and deadlines (which may either be listened to, or just get you fired), or simply ignore the deadlines altogether because they're made up and follow the best practices anyways? To me, it feels like there are no good options if you plan on staying at a company like that.
> I could have tested it better, but that would have meant working late into the night. No thanks. I had already worked a few late nights/weekends and I was done.
Regardless, it feels like you should put your health first, always. So, in a situation like that, consider the potential risks of what could happen due to badly written code, how likely those are to occur if you don't double check your work and if it feels like it's a good idea to check it but you don't have the time, just do it the next day. Of course, this may or may not play into the best practices and getting fired bits above.
On an unrelated note:
> b) Some project managers and Important sales managers had also been doing the same thing – making up phone numbers and locking real phones. Of course, they were doing it manually so only locked 3-4 phones, while we locked 10,000, but the final effect was the same. We were all pissing in the pool. Everyone pretended it was an honest mistake. Lol, look how silly we are. Not a big deal, live and learn ole chap, live and learn. Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him.
This feels illegal. It probably should be illegal. Is it?
kinda unrelated but can someone explain to me how this one korean company has access to infrastructure to lock any phone anywhere in the the world? am I misreading something?
Putting aside the issue of locking real phones this has an issue of state persisting between test runs. The previous execution of a test should not change state for subsequent execution.
This is a long way of asking why the phones weren't unlocked as part of a cleanup step in the same test. Although, at this company, if they did this, they might not notice they were locking and unlocking phones whenever this ran and then periodically disable and enable their customers phones.
Right at the beginning when writing your script you must have known that the random generator would end up creating valid numbers? Expecially when run for tens of thousands cases?
There exist test ranges of numbers that are allocated in order to accommodate this type of testing and avoid exactly this problem.
Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones.
1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manager had been fired for not being fast enough, and the whole test department was at risk of redundancy. Vice president level executives were asking for updates in daily standups.
Like I say in the post, sometimes, you do what you're told or you find another job. If it hadnt been the height of Covid , I might have chosen option 2.
2. Why didnt we test on a staging / dev server? We did have those, but they didnt work with the K-Pop companies servers. And we didnt have time (or enough people) to make it work. Lots of other engineers decided to just quit. Besides, we never thought K-pop would actually lock phones we didn't own.
3. Some people have asked about it: This type of app is perfectly legal. If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. This is builtin on iPhones, Android need external apps to enforce this.
4. While it may seem stupid in retrospect (LOL, this guy calls himself an engineer, tests in production)-- at that time, it was just one firefight after another, and there was no time to think or plan. I can joke about it now, but at the time, it was one of the worst pressure cooker environments.
Some updates: After we released the 3 products, the whole QA/test department was made redundant. Including my boss, who was fired when he was attending his mother's funeral. Funnily enough, I stayed because I knew details of a legacy product. So far the last 3-4 weeks, I had no boss or anyone to report to.