If true, this is a pretty big blow to the viability of RoR as a large-scale platform, at least as it is right now.
I've always thought RoR's tight coupling with traditional SQL would be a serious scaling issue. RDBMSs in the traditional sense just dont scale easily. Google figured this out a while ago (hence BigTable). And RoR reads and writes soo much that you are left with a burden on the DB that might have been better mitigated by doing more of the processing client side and persisting with a simplified cloud-based service rather than a single, central RDBMS.
Not to stir up a hornets nest, but I really think this is a limitation of RoR that should be addressed in the future. Abstract out the SQL so other methods of persistence are possible.
EDIT: I am not trolling RoR. I am simply proposing that it might be better to be able to use it with something other than traditional SQL for persistence of data, so that traditional RDBMS can be replaced with a simplified cloud-based service such as SimpleDB.
I speculate this might be possible now using Ruby -> JRuby -> JPA -> SimpleJPA (or any other MySQL/SimpleDB wrapper) -> SimpleDB. I dont want to even get into how much work that would be though. It needs to be easier I think.
Rails, and the alternatives mentioned don't really make sense for a messaging application. You wouldn't write the back end for Gchat or AIM in Rails (or PHP, or Django), either. Rails would be fine as the web front end that talked to another messaging system. I've interviewed with the twitter guys and been to their talks and they definitely are NOT doing the right thing. However, "the right thing" is not easy and there are few people around and available who can do it.
edit: their best bet is to try to find someone who is burned out from working on wall street applications.
Twitter isn't a messaging application in the traditional sense (as in IM or phone networks) because it doesn't involve real-time synchronous communications. All communications in Twitter are asynchronous as in most database-backed webapps and I think it should be possible to scale Twitter with Rails assuming you know how to scale the database tier.
Having said that, it would still be fun to rewrite Twitter in Erlang... :)
Dang - I was going to say Erlang. This is an ideal application for Erlang... scalability and uptime. There's a reason it's heritage is in the teleco industry.
I went and had a look at the slides for their 'scaling twitter' presentation, and it looks like they had a look at Erlang and reacted with some silly "ooh, it looks ugly" sort of comment. Seems pretty silly to reject something out of hand that is very much along the lines of what you need.
The scaling problems have little to do with rails. They admitted their problems were at the database level eons ago. I imagine "the switch" has more to do with the new chief architect switching to his favorite horse. Two whipping boys for the price of one: the old platform, and the old architect.
The new rewrite will give the new architect time to vest, until it fails, like big rewrites usually do. If you're not sure how to fix things, the big rewrite is a perfect solution, really.
The rails DB integration lets you use the database directly whenever it suits you. At that point it's more a matter of arranging your data so that it lets you work around the key pain points. Since you can access the data directly when needed in rails, how will a new framework help if your problem is in the database?
I'm sure there are enough skilled C hackers out there; but for them to bring in a new face to re-architect the system now (which is to say, after all the scalability problems have started) seems risky.
i am really surprised how facebook, so quikly found the skillset to build a 10,000 server site and how come, twitter didnt! just why ...
its the skillset, but again why was facebook successful in aquiring the skillset when twitter wasnt! ... why is twitter considering a rewrite and facebook didnt ... they just grew !!
According to various scalability articles on FaceBook, they continually rewrite their software. It's probably been rewritten far more than Twitter has, you just never hear about it because TechCrunch doesn't have a source inside.
In my experience, that's really the only way to do software. It's completely unrealistic to think you'll get it right the first time, and yet completely destructive to go and start from scratch. So you pick your worst problem, rewrite it, then hook it back into the working system without touching the rest. Then pick your next-worst problem, rewrite it, etc.
For me, Facebook often seems very sluggish and slow - I have a pretty useful 4Meg connection and others sites are fine - maybe 10,000 servers is part of the problem - it seems like a ridiculously high number. I wonder how many transactions/pasge views per second they get ...
The bottleneck/stress-point is almost certainly the unique IO patterns and latency expectations. I doubt CPU is the issue or that C is necessary; instead they need to figure out the right fanning/batching algorithms, and buy gobs of RAM and use it efficiently.
With their recent fundraising, they might even be able to run a couple parallel fixup projects in competition, and let the best win.
My knowledge of Facebook's chat architecture comes from drunken conversations with Facebook engineers at bars. So no, I don't think I know any more than that. hiccup
Do you obsessively check the headers for proper status codes whenever you land on a page? I'm always amazed how people find out header problems like this just by browsing.
Michael never claimed to be a journalist. You're likely mixing up the professions because you think TechCrunch has gotten so big that it has and cares about its reputation. That wouldn't be the case, he just writes whatever he can to continue to generate pageviews.
Do you and Arrington have some kind of confidential knowledge about Twitter's implementation that allows you to conclude that its problems are directly attributable to Rails itself and not how it was implemented, or something that has nothing to do with the programming language at all (the hosting facility they use, their operating system, ...)? Or are you just trolling?
But I don't think it's ridiculous to assume that the stability and operational functionality of a website is primarily dependent on the web framework the website uses. The data center is probably the second to blame, but I don't think that Twitter's problems are related to continual issues with their hosting (though i'm sure some are). I doubt the blame lies on Debian, or whatever their using.
It is possible to write buggy, unstable, unreliable, unscalable applications in any framework or language and just because you use a certain technology doesn't mean anything about whether you'll create an app with a good experience.
Handing a sniper rifle to someone does not make them a sniper. Snipers have personal qualities and training that make them what they are; their weapons are just tools that help them realize their inherent abilities as well as possible.
Why are you and other TechCrunch people so eager to assign blame when you don't actually know anything about what you're assigning blame for?
Because TechCrunch thinks our industry is so great that it deserves its own National Inquirer. Seriously, these people should be marginalized and put in the spotlight for the trolls that they are. I'm so sick of the bottom-feeding rumour mongering that scrapes by as journalism these days. Seriously people, stop reading TechCrunch and go checkout a book on O'Reilly Safari or something...learn something worthwhile with your spare time.
"Handing a sniper rifle to someone does not make them a sniper. Snipers have personal qualities and training that make them what they are; their weapons are just tools that help them realize their inherent abilities as well as possible."
This is a good point. But, in the interests of balance, the twitter situation could also be compared to arming a sniper with a broadsword and then being surprised he fails. The choice of analogy makes a surprising difference in how you think about the situation.
"Why are you and other TechCrunch people so eager to assign blame when you don't actually know anything about what you're assigning blame for?"
I would rephrase the question to "what do you and the other tech crunch folks know that gives you confidence in your statements about decisions at twitter".
Point being no one knows what the folks at tech crunch know (or not).
Scalability problems are usually endemic to a particular setup. There is always another bottleneck. The way you're viewing scalability is different than the reality of it.
My previous company (a web testing company) did a load test for a large tax return company. When testing started, their app could support 3 simultaneous users. When the bottlenecks were removed, one by one, the same app (framework, codebase, language and all) supported north of 10,000 users. As you test, there is usually something like a database lock problem, high cpu from a loop that needs to be optimized, bad query/data layout, memory usage maxing out, or something you are running out of, one way or the other.
The people who point out that twitter is not a standard web app and might be better served by messaging have a point also.
What the parent says is that a proper assessment will require an in-depth knowledge of how it is actually implemented. Taking random shots like this article does is like a blind man trying to grope an elephant.
I would like to second the call for an immediate stop to elephant groping.
Elephants are a dear and treasured species. This continual assault by blind people and various other gropers have too often left our Loxodonta Africanan friends cowering in terror.
As for RoR, I'll simply quote a good friend of mine. "RoR rocks! Scaling isn't that big a deal. Took us over six months to get the stack just right for a recent customer."
I'm embarrassed by a lot of comments here. I thought people on this forum would know better. It's like I'm watching a car, with smoke and flames pouring out from under the hood, pull over to the side of the road and people running up and checking if maybe it needs new tires.
You could probably rewrite the front end in applescript and you'd see little difference, and if you did just throw more servers at it. The problem is probably that theres a DB at the heart of all the message crazyness, you can't do that and expect to scale.
I am Ruby learner. The way ruby works with DB is problem for novice. If programmer don't do careful coding for caching then it is bound to cause problem. Ruby hype is all around auto code generation and simple coding/scripting. But when you are writing enterprise level apps then there is more to consider.
Totally amused he threw Zed Shaw in there. From what I remember, Zed trashed several members of the rails core team, not the framework itself or even the entire community.
Surely they are just rewriting the slower core areas of the API in something faster/more distributed. There is no reason they'd have to rewrite the entire thing from scratch.
Zed certainly did trash the community. Take a look at
"This is exactly what makes Rails a ghetto. A bunch of half-trained former PHP morons who never bother to sit down and really learn the computer science they were too good to study in college. [...]
After this, I started thinking, since it was the first realization that I picked the absolute last wrong community to make my living in. They were all pricks, morons, assholes, and arrogant fucks who didn’t care about the art or the craft.
They were all a bunch of little ghetto fabulous princesses trying to make it in this tiny little Rails world and not enough brains between them all to make it happen."
and about the framework
"Now, DHH tells me that he’s got 400 restarts a mother fucking day. That’s 1 restart about ever 4 minutes bitches. These restarts went away after I exposed bugs in the GC and Threads which Mentalguy fixed with fastthread (like a Ninja, Mentalguy is awesome).
If anyone had known Rails was that unstable they would have laughed in his face. Think about it further, this means that the creator of Rails in his flagship products could not keep them running for longer than 4 minutes on average."
Although he does add that
" [...] Rails is a pretty nice idea, and it demolished the Java world I hated so much."
It shouldn't be an all-or-nothing situation. There are obviously parts of Twitter that would work best as non-Rails. Ruby is too slow for certain things, and so you determine which of those certain things are in your application and you build them in C or Java.
For Twitter, I think some parts of the API need to be in C. But the whole service? I doubt it.
Twitter has components in several languages (even erlang for xmpp)
They say most of their traffic is API requests, and I would expect that they would want to optimize that porition of their system. Rails is composed of many components, many of which you don't need for a REST service that talks json/xml. Blaine and other twitters have talked about their experimentments: other app servers (Thin), queueing systems (before building starling), ...
I seriously doubt they are actually going to rewrite Twitter in another language. That's a pretty dumb statement and prediction, and Michael Arrington just lost another notch of respect from me (he lost the previous one for his vicious attacks on that Blaine dude who archtected Twitter).
Sounds to me like Mr Arrington has an axe to grind and is putting that ahead of objective journalism.
I assume the difficulty is storing the data and then retrieving the right data when generating a page. Yeah, you can use a SQL database, but that doesn't seem to be working very well. So they probably need to write some custom storage backend that is actually designed for the load that is put on it. Relational databases are general solutions. General solutions are good for general problems, but not necessarily specific problems.
Anyway, if the problem happens to be that the Rails frontend is too slow, speeding that up isn't going to buy them anything long-term. Eventually the database is going to bog them down even if the web GUI is hand-optimized assembly.
JMHO, obviously I have no idea what the twitter codebase looks like or what specific problems they are having. I also don't use Ruby or Rails, but I doubt that's their problem.
That's the theory. Sometimes the practice isn't so simple, when you have 32423432234 lines of spaghetti code, all tangled up. Surely twitter isn't that bad, but I've been in situations where someone cited that very idea and it just wasn't feasible.
Unless you have massive engineering resources that's not dedicated to feature enhancements and new developments of your main product (ie. a small startup environment), a full rewrite is just too expensive.
Regardless of the reason, if something doesn’t work for you, you need to identify the root cause of the problem and don’t be too arrogant to move forward even if it questions your core beliefs
Well, yeah. What would prevent a language from scaling?
The reason you'd ditch Java or PHP is because they are lacking a lot of higher-order constructs. (PHP epsecially. Statically-scoped variables? First class functions? An object system that isn't a hilarious joke? Unicode support? PHP doesn't have 'em.)
I've always thought RoR's tight coupling with traditional SQL would be a serious scaling issue. RDBMSs in the traditional sense just dont scale easily. Google figured this out a while ago (hence BigTable). And RoR reads and writes soo much that you are left with a burden on the DB that might have been better mitigated by doing more of the processing client side and persisting with a simplified cloud-based service rather than a single, central RDBMS.
Not to stir up a hornets nest, but I really think this is a limitation of RoR that should be addressed in the future. Abstract out the SQL so other methods of persistence are possible.
EDIT: I am not trolling RoR. I am simply proposing that it might be better to be able to use it with something other than traditional SQL for persistence of data, so that traditional RDBMS can be replaced with a simplified cloud-based service such as SimpleDB.
I speculate this might be possible now using Ruby -> JRuby -> JPA -> SimpleJPA (or any other MySQL/SimpleDB wrapper) -> SimpleDB. I dont want to even get into how much work that would be though. It needs to be easier I think.
Startup idea maybe?