instead we buy and set up an entirely new set of servers
What property do you work on? How is it more efficient to buy or reallocate entirely new servers than update your old ones?
How many times per year do you even update your application? The cost of getting the servers ready and installing packages would be prohibitive if you had to roll out, say, an urgent security fix.
I think it depends on the kind of upgrade. Seldo is describing a major migration to a new platform. In the case of a point release a rolling migration or the type I described above are the norm.
When doing a major release seldo is spot on. Especially given the small amount of time it takes for hardware to get out of date it's often cheaper to provision new hardware prior to the major release and then deploy straight onto that. Mostly the old boxes would then get cycled back in to production, something less strenuous like staging, or retired.
Exactly right. As I said at the beginning of my post, the "new boxes" method is only used for major platform upgrades, of a magnitude that would happen once a year or less for most properties.
For the really huge properties, the cross-colo upgrade pattern you (sh1mmer) describe is the way to go.
Ok, I see what you are saying. Maybe it makes sense if you were switching operating systems or were upgrading PHP from say 4.x to 5.x. I'd still like to see the numbers behind it though.
Also, cross-colo is not a factor of being big, it's a factor of having a BCP (business continuity plan). Every team at Yahoo is urged to have BCP for practically everything. I used to do cross-colo upgrades for a relatively small project once we had more than two frontend servers.
I think cross-colo is a size thing. BCP is good for all business, but most start ups can't afford to pay for hosting in several data centers.
I image cheap way to fake multiple colos now is to use visualization, amazon style. Heck you could do exactly what Seldo was suggesting every time on EC2.
Widgets, Profiles, YDN and Mobile, in the last 2 years. And those are just the ones that I know about because I worked on them. YAP was also new hardware but it was also a new product.
Also, the new Buzz site used all new hardware (and was a new product) and Games rolled out on new hardware when it was upgraded from HF2K to PHP. The old servers were reimaged and folded in with their PHP siblings once it was clear that the build was holding up properly.
What property do you work on? How is it more efficient to buy or reallocate entirely new servers than update your old ones?
How many times per year do you even update your application? The cost of getting the servers ready and installing packages would be prohibitive if you had to roll out, say, an urgent security fix.