Hacker News new | past | comments | ask | show | jobs | submit login
Apple Confirms That Its Dev Center Has Been Breached By Hackers (techcrunch.com)
197 points by tomashertus on July 22, 2013 | hide | past | favorite | 128 comments



I am very surprised the root cause of the incident wasn't mentioned by any other HNers

This is a serious exploit on Apache Struts 2, a popular Java Web framework known as SSH (Spring, Struts, Hibernate). If you visit many websites, and see URLs ending with .action , it's probably written in Struts 2

S2-016 / CVE-2013-2251, effecting Struts 2.3.15 or lower

http://struts.apache.org/release/2.3.x/docs/s2-016.html

The exploit was deadly easy to weaponize as the Apache folks blatantly published PoC on their own bulletin

Example of server side arbitary Java code (OGNL expressions) execution

http://wp3.sina.cn/woriginal/761d2801jw1e6pqfs8hrbj20c70gomy...

Apple.com hacking was published on a popular Chinese security bulletin website

http://www.wooyun.org/bugs/wooyun-2013-023444

The issue was submitted to APPL on 2013-05-10 but ignored, and went public on 2013-06-24.

A Chinese blog on history and technical details of the exploit:

http://www.inbreak.net/archives/507

As a side note, rumor is that about 60% of Chinese goverment, e-commerce, banking, gaming websites was hacked using this S2-016 exploit, database was dumped, and exchanged in underground market. Also past records shows that the Apache Struts team is incompetent at security:

http://taosay.net/?p=611 (Warning: rant in Chinese text)


So Struts 2.3.15.1 which has the fix for this was released 16 July 2013. In fairness, it may be that Apple, being as big a target as they are, had little time to react before they were penetrated. But this really goes to show that when you are informed of a "highly critical" remote code execution vulnerability in one of your public-facing applications, you need to drop what you are doing, take the service offline immediately and start the process of upgrading/patching. You may literally have only minutes.


This reminds me of the post I saw on here, can't remember exactly what it was called, but the guy talked about putting servers up with a honey pot and at this point within hours they're getting scanned and probed.

He said it used to take days or weeks for that to happen. Now it's hours.


I used to work for the kind of companies that only use Struts+Spring+Hibernate, and it's simply appalling how many of their applications are running on year-old library versions with severe security flaws. One week is the median lead time for an emergency deployment.


It was mentioned on this thread, 2 hours ago:

https://news.ycombinator.com/item?id=6082212

This link points to the top comment on another thread (which has more points and comments as of now) about the Dev Website outage.


yeah but the reaction was vastly different.

I remember the last RoR exploit was quite a thing on HN few months ago. But not this Struts 2 one.

Tip: if you see some URL end with .action, it's propably Struts 2 and vunlerable to S2-012, S2-015 and S2-016 such.

Besides, apple.com was hacked as early as in May using similar Struts2 exploits.


I don't think Struts is nearly as popular with the HN community as RoR


IIRC when YAML exploits were announced it was quite popular to call rubyists incompetent amateurs.


Stating and/or implying that others are less competent than our idealized selves is what this community loves to do most. Any particular tech is incidental.


Can't believe how easy that exploit is


What I can't believe is that an expression language that so directly interoperates with Java objects is processed from request parameters.

I can't conceive of the specific use-case that justified adding this, but it's so clearly a bad a idea that it doesn't appear bad only in hindsight. It appears bad in foresight.

"I'm going to accept and execute arbitrary expressions from clients, and these expressions can interact with arbitrary Java code, is that cool?"

"No."


As the person who made that decision some time ago[1] I can tell you that basically we didn't fully appreciate or understand all the features OGNL supported at the time.

As we realized it could, say, invoke static functions and execute extremely complex expressions, we realized the hole. Since then the team has been patching them piece by piece, but really OGNL should be thrown out and replaced with something that is far more limited in it's capabilities.

It's been a long time since I was an active contributor, but that'd be my recommendation if I was still hacking away on it.

[1] https://news.ycombinator.com/item?id=6081428


>Also past records shows that the Apache Struts team is incompetent at security

Compared to Rails, PHP, what?


I don't think there is any evidence that the hack attempt was linked to this vulnerability. It seems more likely that it was the Turkish hacker.


Not giving any notification for several days was pretty bogus. Probably not actionable under CA law, though.

The sad thing is I know several competent people who work at Apple on security, although I think they do OS security not online-services. Apple has proven over and over again that they're utter crap at online services. They're going to be totally pwned at Blackhat w.r.t. iCloud in a week, too. I don't understand why they can't just drop $500mm to $2b on buying one or more competent saas/ops companies to get some real expertise in house, rather than relying on 15 years of accumulated contractor/vendor built crapware.


>Apple has proven over and over again that they're utter crap at online services.

Yes. They only have the most successful online app and music store service on the planet.

And one of the biggest backup online service (iCloud) too.

Oh, and the most popular online computer shop.

Utter crap indeed.

>I don't understand why they can't just drop $500mm to $2b on buying one or more competent saas/ops companies to get some real expertise in house, rather than relying on 15 years of accumulated contractor/vendor built crapware.

For one, you have no idea what saas/ops people they have in-house. Second, you have no idea how their systems are setup.

Second, you have this baseless idea that throwing money at an engineering problem solves it (yeat, it worked great for Brooks, Mitch Kapor, and tons of other multi-hundrend million failed projects out there).

Third, who told you it's "15 years of accumulated contractor/vendor built crapware"? From the little we know, their current foundation is a cloud on top of Azure. Which is anything but "accumulated".


He wasn't saying that their online services aren't big or aren't profitable, he was saying that they aren't very good. And I would have to agree with him on that. I don't think I've ever heard anyone argue that either iCloud or iTunes are good pieces of engineering.


Seriously. Miserable load times in iTunes, not being able to use the same Apple ID to access different teams in iTunes Connect, lack of an API to the dev portal, lack of OAuth support, laughable reporting in iTunes connect, cryptic sync errors with iCloud. The list goes on and on.


They fail at even the most basic stuff too.

Like how when you launch a new app on the App Store, for the following 6-8 hours users get random errors ("not available on store", "not available in this country", sporadically missing from search, etc) when they try to download it as it propagates through the app store CDN. This even affected the recent OS X updates launched on the Mac App Store. Imagine if Amazon CloudFront worked like that!

And imagine if Amazon had to shut down their whole site when they updated one product like the Apple Store Online.


> And imagine if Amazon had to shut down their whole site when they updated one product like the Apple Store Online.

Most of that is just hype machine on Apple's part. I doubt there is a technical reason to do so.


It makes sense to shut it down during keynotes, events, etc. But the store also goes into "we'll be back" mode once or twice a month for routine updates.


>> Apple has proven over and over again that they're utter crap at online services.

> Yes. They only have the most successful online app and music store service on the planet.

> And one of the biggest backup online service (iCloud) too.

> Oh, and the most popular online computer shop.

> Utter crap indeed.

I really don't understand this argument. The quality of the code powering something is completely orthogonal to the amount of money said thing can generate. As long as the transaction goes through, money can be made. That doesn't mean that behind the scenes it's not a complete disaster.


>I really don't understand this argument. The quality of the code powering something is completely orthogonal to the amount of money said thing can generate. As long as the transaction goes through, money can be made. That doesn't mean that behind the scenes it's not a complete disaster.

Completely orthogonal my ass.

Where does any idea about the quality of code comes from? From an operational point of view, not only it WORKS, but it works in a CRAZY scale.

Hundrends of millions of customers with hundrends of millions of credit cards, serves multi-TB of stuff every day, handles music, video, apps, updates, etc. (Oh, and it's not even that which is down -- it's their developer portal).

So where does the ideas about the "quality of code" come from? Have you SEEN the code?

If not, we just have the "works, serves more than half-a-billion, at a Facebook like scale, with more credit cards than even Amazon" to go by.


It works, at a huge scale, and poorly.

Post a new app to the App Store. Poll various friends to see when they can actually access it. You can watch their database replication slowly happen in real time over the course of hours, as the app randomly appears for more and more people. People will often be able to access the app through a direct link but not through search for a few hours before everything synchronizes. You have to give it about twelve hours before you can reliably count on all users being able to access it.

Let's compare this to, for example, Google search. The scale on Google's end is much larger. What's worse, they're indexing external content that they have no control over. Despite these handicaps, they have no problem with rapid, coherent updates. More than once, I've typed up a reply to a comment like this, posted it, then hit up Google for some additional information and found my own comment posted just minutes ago among the top results.

Google, working with external data they don't control, serving vastly more requests of vastly more complexity, is able to provide coherent results within a few minutes, while Apple takes hours to roll out new data that you explicitly send them.

That kind of stuff is where ideas about code quality come from. No amount of "but they make lots of money and are popular" can counter this.


I would say that while you're correct about Google being faster the reasons are probably less obvious.

Whereas Google is going for absolute fastest delivery of content, Apple is trying to deliver a full ACID database system with guarantees their licensing vendors will sign off on. This likely means massive replication and it likely means that Apple is enforcing a lot of service guarantees to ensure these are atomic transactions.

On the other hand, Google just really wants you to have blisteringly fast load times. Different problem domain in my opinion. One is an index with few guarantees necessary, the other would not be obliged to sell you content if one of its guarantees failed. Different strokes for different folks.

Granted Apple could be faster, but I don't think it's fair to compare their propagation to Google's.


I don't understand. Apple needs to make a lot more guarantees about integrity and atomicity, which is why their data integrity sucks and their transactions are not atomic?

I could kind of sort of understand if apps took twelve hours to show up on the store, in general. But no, they take twelve hours overall to show up after a long period of bizarre, inconsistent rollout.

The speed difference is much less important than the fact that Apple presents its users with a wildly inconsistent view of their database for any recent changes for a period of hours. Of course, updating changes quickly would be one way to fix this, but the speed is not in itself the problem.


>It works, at a huge scale, and poorly.

If it manages to push 1 billion apps and 1 trillion notifications, billions of songs, TB of video and such, with no major complaints other than "it takes 12 hours for an app to appear for everyone" that's doing great in my book.


Well, it seems we come down to a pure difference of opinion. We appear to agree completely on the facts, I just happen to think those facts means their operation is fairly poor, while you think it's great. So, we can't do much now but move on....


The standard car's internal combustion engine works at a crazy scale, however it still dumps over half of the energy of the fuel as heat through the radiator. Things can work pretty badly and still scale.


If it's as good as the "standard car's internal combustion engine" then it's mighty near great.

Seing that the other alternatives to "standard car's internal combustion engine" are either off the market or marginal/niche.

People complained in this post that it wasn't good. As in: crap compared to the standard.

For the analogy with cars to hold, it would have to be like a seriously brain damaged internal combustion engine -- not like the "standard car's" one.


If we're going with cars, it's more like say, 1960s British cars. Amazing industrial design, good engines and overall performance, etc., but crap electrical systems compared to other companies of the time -- yet people bought them because the overall package was still good, often largely due to how amazing the E-Type looked. Saying "the Jaguar electrical system sucks; for a company who makes such amazing products it's an embarrassment" was perfectly true, even if you still buy a Jaguar (along with lots of other people).

(although I think it was mainly other British marques who had horrible electrical systems at the time, and was more a 1950s thing)


We know what frameworks and other tools they use. Those suck (which is a personal evaluation, sure, but the marketplace has pretty much validated my opinion on this). The performance/feature characteristics visible to the end user also suck.

Apple's developer and support forums are pretty worthless compared to other vendors. A lot of information is missing or hidden, and finding anything worthwhile requires a lot more clicks than finding something at Google.

The whole iOS approval process is...Kafkaesque.


>We know what frameworks and other tools they use.

We only know about SOME of them (e.g WebObjects).

>Those suck (which is a personal evaluation, sure, but the marketplace has pretty much validated my opinion on this).

In the same way that the marketplace has validated DOS and then Windows over UNIX? Javascript over LISP? VHS over Betamax?

Actually, WebObjects was one of the finest web frameworks (including in it's Java incarnation). Sure, there are newer things now, but nothing extremely better. Not to mention even giants like Facebook and Yahoo use PHP for christ's sake. It was also one of the most popular in its range, when it was available. It just didn't make sense for Apple to participate in that market.

>The performance/feature characteristics visible to the end user also suck.

Compared to what? In the same scale? Never hard any real issue with ITMs (at least none that I didn't have with online services 1/10 it's size).


WebObjects. Essentially a legacy from NeXT, almost 20 years old, and even Apple abandoned it for everyone except internal use. Finding competent people for that is...nontrivial.

They also use a lot of Sun/Oracle in general (especially in the early 2000s). Look how well that worked out for eBay.

Given the runaway success of the iPhone for hardware reasons (and I guess iOS, and third party developers), you can't really claim the success of the App Store is due to the quality of the App Store. The iPhone was successful first, then demand for apps, then Apple built the App Store once regular people were jailbreaking their devices and doing their own development.

I'll concede iTunes was successful for music on its own, but that's more for licensing reasons than anything else; I find what.cd a vastly superior experience as a user, even independent of money, and Gazelle/BT are open source. Both are better than what the RIAA came up with for sure. Arguably Spotify, Rdio, Pandora, etc. are better.

Netflix has done a way better job on video than Apple, too.


"Given the runaway success of the iPhone for hardware reasons..."

I wish there were more folks who could see that. Your comments about Apple on this thread are worth contemplating.

There seems to be this silly idea that Apple is now "more than a (proprietary) hardware company". But if we took away the design-patented, hermetically-sealed enclosures, what is the value of Apple? What's left that we can use? Can any of it stand on its own?

It should be obvious, but only after you tie users to a particular piece of hardware do the other opportunities like playing middle man to digitized consumables like music, software or books arise. I surmise it is those opportunities that cause people to believe Apple is more than a hardware company. But maybe Apple is just a hardware company that, outside of their area of expertise - hardware, is very opportunistic? "Good artists copy, great artists steal."

How many times have we seen Apple "borrow" from third party software that supplements Apple's OS functionality and proceed to incorporate others' ideas into their next OS version?

Are all the components of Apple's allegedly diversified business dependent on the success of Apple hardware? Try the following thought experiment:

Subtract the fact of the Apple hardware with its attractive form factor and imagine other companies designing the world's most popular mp3 players, phones and tablets. Imagine Apple does not make, sell, or have made hardware enclosures or the cheap, Chinese-made electronics that are hidden inside. Now, ask yourself, "What is the value of Apple?"

The value of iTunes without Apple hardware?

The value of the AppStore without Apple hardware?

The value of Apple's latest Mach/BSD hybrid OS without Apple hardware? OK, but how easily can Apple's graphics layer run on other hardware?

...

Filemaker Pro for Windows? Eureka!

Yes, Apple is much more than just a hardware company.


Did you just compare a BitTorrent tracker to iTunes?

Anyways ignoring that, the "App Store" was built alongside iPhone and released at the same time. So I'm not sure where you're getting yo facts from.

But Netflix has done a much better job on video. And I agree that iTunes is a little weak.


Third party apps on the iPhone were originally supposed to just be web apps run in Safari. iTunes was just for the loading of music (and I think video?) and basic activation and such. The iPhone was announced in January 2007, and released in June 2007. The SDK was announced in October 2007, released in early 2008. You had to wait until users upgraded to iOS 2.0 (released summer 2008) to actually run those apps.


No you're right... Still, I find it hard to believe that they built the entire iPhone SDK and App Store in less then a year. I'm going to guess it was part of the original idea, just delayed for some reasons.


Of course they didn't build the SDK in less than a year. It's a pared-down version of what they use to create their own apps.


The App Store and iPhone 3G were launched a year after the original iPhone. That is also when the SDK was released and 3rd party apps were introduced.


> Oh, and the most popular online computer shop.

No, they don't. Amazon is bigger.


Also the pure online hardware buying part is by far the weakest part of the argument.

The Apple Retail Stores are innovative (sort of; not that uncommon outside of computers, even in design, but bringing that to computers, sure). The Apple Online Store is pretty boring and simple -- limited SKUs, wasn't international for a fair bit, etc. The backend infrastructure to customize orders and manage manufacturing is somewhat less advanced than Dell was. It's just a question of making amazing hardware that people will do anything to buy. I don't think a random startup would have a hard time with the online store portion.


For their computer sales only?

I don't think so. How many people buy computers off of Amazon?

But even if so, that's a negligible pedantic correction. It's still in the top two.


> why they can't just drop $500mm to $2b on buying one or more competent saas/ops companies to get some real expertise in house

I don't know. Look at how huge iCloud and the iTunes store are. Every iOS device sold uses those services. They'd need to buy twitter or facebook to get people that are experts on that scale.

If you take the scale into account I don't think that their web services are 'crapware' - especially for a company where those services are only complementary to their main products.


It doesn't have to be a consumer service; they could buy Akamai or really any decent sized CDN, any of a bunch of network carriers, managed hosting providers, etc. Apple's stuff is all really easy to shard/partition, so it's nowhere near the technical challenge something like Facebook is, where it's largely one big monolithic thing with lots of interconnection. (I wouldn't be suggesting Twitter as a source of amazing ops/security talent, in particular.) Or they could buy Canonical or Red Hat or SUSE, or to stay BSD, iXSystems. Twilio or Github also probably have a high level of ops/tech competence vs. cost to acquire, even if they're dealing with a smaller userbase currently. Valve would be another obvious one.

Jobs was right -- in 10 years, iCloud is going to be central to the success or failure of Apple. There's no way I'd pick Apple based solely on iCloud so far; it's markedly inferior to what Google has put together, and actually basically inferior to what a modernized RIM BES could have been.


Canonical? To learn from the experience with Ubuntu forums?


Yeah, that one's weak -- I wanted to say Debian, but they're not a commercial company.

As far as I'm concerned, the gold standard for developer relations, including high-dollar/enterprise customer support, (at least at a large company) is Microsoft. It would probably be politically infeasible for Apple to license MSDN though.

Which is weird, because back in the 1980s (and maybe early 1990s, still), Apple was pretty awesome for developer relations. I only faintly remember it (I had Macs from 1985 at school, and from 1990 at home, and only did the most basic of stuff with ResEdit and checked out the books from the library), but stuff like their UI/UX guidelines were way ahead of their time.


>> If you take the scale into account I don't think that their web services are "crapware"

What "scale" has to do with "crap" ? Considering both vertical and horizontal scaling - there is possibility that "crap" can be scaled too.

http://en.wikipedia.org/wiki/Scalability


you sound ridiculous. sorry but you expected better than a 4 day turn around on this? pull your head out of your ass. these guys are probably still pouring over logs, doing forensics...


I don't expect them to say what it is, definitively, for a month or so.

I do expect them to give some basic initial notification to developers that a security breach has happened, it was limited to x, y, z (which they should be able to determine within a day), and what initial steps are, and have some independent way for developers to contact them.


Has Apple ever made an acquisition on that scale? I don't thin that's in their DNA.


Apple has bought companies in the hundreds of millions of dollar range several times (generally suppliers or tech vendors; treating "better online services" as a form of technology like a fabless semi or fingerprint sensor seems reasonable.)

Due to the price premium for being a "winner", buying a $100-200mm (in a down market) really well executed SAAS company for $1b (now) isn't that big a deal when you're sitting on hundreds of billions and tens of billions a quarter in profit. Online services will ultimately be that important to Apple's success.


Not in the billion range, but they were acquired by NeXT for negative $404m (in 1997), they acquired PA Semi for $278m, Quattro Wireless for $275m, C3 Tech for $267m, Anobit for $390m and AuthenTec for $356m. They are clearly willing to pay hundreds of millions if they think it's worth the price.


Given there market cap at the time of the deal, NeXT was a huge buy.


A "security researcher" posted a video with compromised accounts claiming to have deleted all the data after reporting the bugs to Apple: http://www.youtube.com/watch?v=q000_EOWy80

It is really unfortunate and irresponsible that data in the video is not obscured.


There was no need to take all that data at all. If he was interested in security he could just set up a test account, test the vuln with that account only, and send Apple the results. Job done. I doubt Apple would bother to prosecute for that, they might even thank him. Do they pay bounties for reports on security vulnerabilities? If not they should.

Taking all the data and publishing it, and then bragging about it on youtube (!) just leaves him open to prosecution, and I imagine Apple will go after him now for publishing the data of their developers.

I'm not sure I understand the logic of publishing this, but there seems to be a mentality of braggadocio among wannabe security researchers -

1. Hack high profile website and publicise it

2. ?????

3. Profit!

I'm not sure what step 2 is.


I don't really understand the amazement of everyone after looking at this video. He's Turkish (I think), not bound by U.S law and published this video as some kind of demonstration of his capabilities as a security consultant hoping to secure a nice gig or two.

Not the brightest idea he had there but neither are the responses here auto-assuming that if the video is in English then that person must be bound by U.S law.



He's bound by any U.S. law covered in the U.S./Turkey extradition treaty.


Generally speaking countries don't extradite their own citizens.


I wonder how much of the problem is due to him being foreign (Turkish?) and neither speaking English as a first language nor necessarily being immersed in the technical/legal culture of the US. There might be a use for some kind of legal/PR advisory organization for people who want to do "aggressive disclosure" or "full disclosure" instead of "responsible disclosure", while not being proper blackhats with anonymity and lulzery.


Wow. I'm speechless. This guy is really stupid or doesn't care. He can (or rather "will") get into serious legal troubles.


Agreed. What he did might have worked in 2004. But these days he's just going to end up in serious legal hot water.


I'm not 100% sure, because he didn't even blank out the emails.


How is this legal? Seems like someone should be facing charges for this...


The Computer Fraud and Abuses Act is incredibly broad, it's basically impossible to prove a vulnerability like this exists without violating it.


It's of course NOT legal … I've just reported the video to Google.


It's amusing (and sad) that while the NSA monitors all emails and every single phone calls of all Americans, but still fails to detect and prevent a computer intrusion of this magnitude into Americas largest company.


I've just reported the video to Google sounds like a dirty, dirty euphemism


The video got taken down, is there a mirror anywhere?



Actual comment is here: http://fyre.it/tjlVmC.4

Says alternately that he's taken 100,000+ user records, or just 73 Apple worker records, or no user details at all. And that he's keeping all the "evidences".


Interesting. Anyone want to weigh in on the ethicality of this? (Either side.)

Downloading 100k userdata records seems quite extreme, but is it unethical for a security researcher to do so?


It is absolutely, unquestionably unethical to do that. There's a huge, huge difference between proving a concept and stealing user data -- no matter what your end goal is. What possible positive outcome could he be looking for in taking this data?


Anyone know if this was illegal, then? Or will his status as a security researcher (albeit one with poor judgement) protect him?


>Or will his status as a security researcher (albeit one with poor judgement) protect him?

It usually doesn't protect you even if you don't take anything.

That's why there's a huge backlash against "responsible disclosure".


There is no such thing as "status as a security researcher". If you want to research security, I suggest you do it in your lab or with consenting adults.


The only legal security researcher is the one hired by a company to identify issues during a security audit.


It's a whole lot safer doing attacks on a device you own (or downloaded software you run on your own infrastructure) -- live pentests on someone else's network infrastructure and hosted applications is pretty similar to a "real" attack.


Penetration "testing" networks you have no permission to use looks identical to a real attack. I doubt Apple will look at him any differently than an attacker if they decide to pursue him.

He refers to Facebook's whitehat list too. Facebook does allow people to try to break parts of their application if they're responsible about the disclosure. To my knowledge Apple doesn't have such a policy so can't have given any implied permission to attempt to attack them.


>> "Anyone know if this was illegal, then?"

I don't know the full details of the case but didn't the hacker weev get imprisoned for downloading 100k user records from AT&T?


He did. It'll be interesting to see how sophisticated this attack is compared to the AT&T one. It looks like there's Java code used to dump this, rather than just using wget.

I think part of weev's downfall was the subsequent conversation he had discussing what he could hypothetically do with the data though.


> Anyone know if this was illegal, then?

That likely depends on the laws of the country he was in when the incident occurred.


The US has argued that they have jurisdiction over any cyber crime which affects a US company or institution, regardless of where the person was - e.g. Gary McKinnon [1] was indicted by a grand jury in Virginia for hacks on computers owned by NASA and the Pentagon despite never having visited the US.

What really matters in cases like these is whether the country in question will arrest and extradite the individual to the US - this is far less clear cut - see the Snowden case for an example of someone attempting to evade US extradition law.

Some countries (notably China) have been fairly opaque from a US judicial perspective - see hacks on Google, the New York Times blamed on the Chinese - which have ended up with no visible action.

1: http://en.wikipedia.org/wiki/Gary_McKinnon


It's almost definitely a violation of the CFAA.


>In order to prevent a security threat like this from happening again, we're completely overhauling our developer systems.

I wonder how out of date it was?

>updating our server software

Why now? This should be done as soon as new releases are available. There's even packages like unattended-upgrades that do it for you.


> There's even packages like unattended-upgrades that do it for you.

Anyone deliberately installing such a package on a production server is dangerous. Their profound lack of judgement disqualifies them from any form of access to servers I control. Updates break things. They get approved by a knowledgable human familiar with the system and installed with suitable engineers standing by, or they don't get installed.


Having out of date packages on production servers is also dangerous. It really depends what you consider your biggest risk to be: downtime vs getting hacked due to an out-of-date packages? I personally would take downtime, but every place is different.


A check for necessary updates should simply be part of someone's regular, preferably daily, routine. It's a basic cost of doing business.

The manpower necessary is not enormous. Any particular security update has a relatively small chance of requiring prompt installation on a particular production service. On the unusual day you're struck by lightning, you pull out the relevant emergency plan and begin executing it.


Remember that a lot of small to medium sized websites are maintained by part time freelancers and don't have anything resembling an ops team, there's nobody being paid to do the day to day running of the servers.

In such a situation it's probably safer to at least have automatic scheduled patching against deadly vulnerabilities and accept that occasionally that might break something.

Of course that wouldn't apply in apple's case.


You're positing a scenario that shouldn't occur in the first place. Such a website should be on a shared or managed hosting service, or alternatively, there are companies that will, for a reasonable monthly fee, perform basic routine maintenance such as this on your servers.

If you want to completely mismanage a server you depend on for your livelihood, I can't stop you. All I can say is you're doing it wrong.


Running on shared hosting doesn't solve this and introduces a bunch of other issues, you still have to worry about wordpress installs or whatever.

There are a lot of poorly managed VPS out there, these would be better served applying security fixes automatically.


I'm sure Apple has the resources to hire enough sysadmins to keep their systems up to date. The question is why it isn't happening.


It looks like the exploit used a vulnerability in Apache Struts that was revealed on 7/16 : https://news.ycombinator.com/item?id=6081428

Given that, your question of "how can we completely avoid zero-day attacks" is nonsensical.


Couldn't you run a custom variant of unattended-upgrades on your dev servers, then if all the test pass, automatically ask a human to pull the trigger on the production servers? In some extreme cases, it might even make sense to let the "all-tests-pass" condition force a security update on the live server.


> tests

Funny man.

More seriously, even if you have complete test coverage of your code, tell me, do you do realistic load testing as part of your automated tests? Does that include making sure the results returned by that load test remain correct? What about verifying that your backup system continues functioning correctly after installation of an update? Your monitoring system? Will your tests catch the fact that your SSL configuration just broke? Do they test your load balancer?

I could go on for a while. My career basically started with production service operations, and it's never stopped being a part of my life since. I've seen things people insisted had to be impossible even while I was staring right at it.

I have a favorite story I sometimes tell people in another context. I once wrote an email that, between my explanations of what happened and the SQL dumps proving it, spanned something close to 10 printed pages. It was, at last, real proof of a bug that I'd suspected the existence of for months, but was told had to be impossible, and couldn't be reproduced.

We were days from deploying a change in production that would have triggered this bug in a catastrophic way, and if we didn't know exactly what was going on, we would have had no way of knowing until customer complaints streamed in.

Guess why the developers and server QA never saw it?

Their machines were in the Pacific time zone. It only affected non-PST8PDT machines.

The bug was a confluence of factors, some of which were in third-party code. If it hadn't existed to begin with, it very easily could have been introduced in a package update, as this particular behavior was not a well-specified part of the package's intended behavior. Automated tests would never have caught it.

And at this very moment, somewhere in the world, there's an HN reader rushing off to make sure all their development and production machines are set to the same time zone.


Wow, my hat is off to you - that's a hell of a bug and excellent reporting when you were on a completely different team. I think this is the reason why, as QA, I have greater affinity to Ops people of all stripes than Dev or PM. Those Ops people who care about their servers or look at the big picture of the deployed environment are a huge multiplier to my testing and understanding of core systems.

Also agreed with your position in respect to the person you're replying to. :)


Server QA can be every bit as valuable to ops, and not just by preemptively finding bugs. At that company, I ended up adjacent to the server QA team. They effectively became an extension of the ops team. Many an hour were spent with one side or the other talking to a disembodied head over the cubicle wall, or with me outright sitting in their cubicle. Emails and IMs were a constant. They saved our asses in the field many times. They also wrote a lot of our usable documentation.

On my way out, I recommended one of them to replace me. Shortly thereafter, he did. Years later, I think he's actually the ops manager now. I wouldn't mind working for him, but political BS drove me out of that company, and pretty much any other company that big.


His credibility as a 'security research' is seriously damaged in my opinion for two reasons:

1. He unnecessarily downloaded so many records

2. He made a YouTube video to brag, showing off names and emails in the process.


I don't think it was for bragging but more trying to cover his ass. But I might even be wrong.


True with his final statement in the video that's what it sounds like he was trying to do. But if he just found the flaw, reported to Apple and did nothing remotely nefarious he shouldn't have felt the need to do the video. It seems to me he knows he shouldn't have downloaded so many records and it trying to backtrack to cover himself.


A security researcher has claimed responsibility in a comment[1] on the article citing his intentions were not malicious and he reported 13 bugs to Apple prior to the dev center being taken down

[1] http://techcrunch.com/2013/07/21/apple-confirms-that-the-dev...


It would be nice if they could knock out some easy wins early. I was trying to set up a new machine on Friday- I wanted to install Homebrew, but it required XCode command line tools. Which are on Apple's dev portal. So I can't use Homebrew (or build anything at all) on that machine.

These tools and other static downloads (SDKs etc.) are clearly not affected by this breach - can't they rehost them somewhere in the short term?


If you download Xcode via the Mac App Store, I believe you can still install the command line tools from the downloads pane in the Xcode preferences. I updated mine yesterday from there no problem. Hope it works for you!


It's available on multiple torrent sites. I just found my self a copy for my newly installed mountain lion.


Bummer, and they have earnings to announce on Tuesday.


The general public won't care so much about the dev center. If iTunes, App Store, or Apple Store had been hacked that would be much more serious from a PR standpoint.


My thought was not that the general public would care, rather it was that institutions own AAPL stock do care, and if they think that this event will affect Apple's ability to deliver product or recruit developers they may decide shift their portfolio's holdings to a different tech company with more upside.


oh shit. Hadn't thought much about that.


There must be some serious mudbutt going on with the upper level executives right now.


Great, and I can't figure out how to change my apple password because it asks me some "security questions" before allowing me to change the pwd (such as "what was your favorite band in high school).


3 days seems like a long time to alert people. Releasing the news early would be the right thing to do but keeping it secret until you know the extent of the breach was probably better for their brand.


Sounds like Weev is about to get some company.


"They waited three days to alert developers because they were trying to figure out exactly what data was exposed"

Why couldn't they notify people about the hack, then alert them to it's specifics in due course? Piss poor excuse IMO.


My guess is that from a PR perspective they might have worried that releasing the information without a definitive scope of impact would have led to days of wild panicked speculation in the media outlets, and userbase.

While there was still wild speculation, security was only one of many possible scenarios being discussed, and it was mostly treated like a regular outage.

I'm not saying this delay in disclosure was "right" (what if it had ended up worse in scope?), but I agree with sibling post (dave1010uk) that it seems to have worked out better for their brand.


Just got a bogus "reset apple id password" email.

> If you weren't trying to reset your password, don't worry – your account is still secure and no one has been given access to it.

Ironic.


Yeah, I've gotten two today. Can anyone speak to the aim of these?


Word is, apns is still up, only the dev center has been effected.


> Credit card data was not compromised

So? Who cares about credit card data - it so easy to just send a list of compromised numbers to the banks and get new numbers. I've been sent new cards at least twice without being told why they are changing the number, except for some nebulous "security measure", so they clearly do do this.

Of all things to worry about credit card numbers hardly rank.

I don't know how it works - but can the breach allow attackers to upload modified apps in the name of the developer?


I would care. If I know that it hasn't been compromised I don't have to get a new number and go around to all the services I'm registered with to update my credit card details. Saves me a lot of hassle.


They are supposed to update those automatically.


Who are? I use my credit card at multiple services. Is the bank going to call around to services that I have paid for previously and tell them my new credit card details? I hope not


When you get a new credit card, only the last 4 numbers will change, and your bank might charge the new credit card for payments that were supposed to be going to the old card if they are deemed low risk (if it's something you used to pay for before)

Of course, YMMV, this is just what my bank does.


Not even close. I work at a large bank and often 12 or more numbers change. Beyond that it is irrelevant how many numbers change. Beyond that only approved but unsettled transactions will go through under the old number. You can't continue to use the old number for new transactions.


> Is the bank going to call around to services that I have paid for previously and tell them my new credit card details? I hope not

Yes. Anyplace that bills you regularly on the old number will automatically get the new number.

> You can't continue to use the old number for new transactions.

It depends on how you define new. A new biller, then correct. But if it's a biller that you have an existing relationship with, i.e. they bill you every month, then they do send them the new numbers (and/or allow them to use the old number).

I can't say if this is a global policy of all issuers, but I can say that I've experienced it. I've also seen it in the fine print when I signed up for repeated billing.


It's usually called something like "automatic account updater" by the credit card issuer. I think it's pretty common for merchant banks and merchant service providers to offer it these days. (We use it where I work.)

Here's an older Braintree blog post about it: https://www.braintreepayments.com/blog/automatic-update-of-c...


People using debit cards would care. It's much more difficult to get payments refunded with them.


> but can the breach allow attackers to upload modified apps in the name of the developer?

It doesn't look like they can. Uploading apps to the app store is done via iTunes Connect. That's always been a separate login from the developer center, and it has not been down during the current dev center outage.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: