The comments here about building search engines that viably compete with Google in terms of quality are hilariously wrong. Hilariously wrong.
The dominating cost is not hiring smart people to work on the problem, it's hiring enough smart people to work on the problem.
Consider. Here at Microsoft, we employ a search relevance staff of -- what, 20% the size of Google's? And Google's engineers are rock solid. MS might be able to make up a small margin, but our search relevance staff -- good though they are -- cannot compete with a brilliant workforce that is 5 times larger than them. If we are going to actually compete with Google, the problem is not hiring smart people, it's hiring 5 times the current number of smart people employed for this task at MS.
Never mind buying such a team. MS can afford that. How do you even find that many people in 2-3 years?
When you consider the rest of the data, the case for building something like Google really begins to look grim. For example, how much does it really cost to build a search engine? We've poured at least tens of millions of dollars into just search relevance (I'm not even counting infrastructure). That's good mileage, considering that this way is littered with the corpses of companies like Cuil, who made investments in this area and failed miserably. Still, while this has been a great deal for us, we're still not there yet, and it's not clear we will be in the very near future. And so it's worth wondering: if MS can't buy something like that, realistically, who can? DDG? lol no.
In the end, this is the true dominating cost of building a search engine: people capital. Other bottlenecks, like engineering debt, politics, etc. pale against the sheer, awe-inspiring investment of Google in people capital.
You know what? I really can't get behind the idea that MS is somehow the stirling example of engineering prowess and discipline. This is the organisation who fails their way through every other operating system. One of the poster children for missing the boat on the internet, tablets, operating systems, cloud services, and more.
I also reject this idea that Google's got a lock on every smart person ever. That they don't have any politics or wasted effort. That they're the snowflake they think they are. This kind of thinking is pathetically misguided, and a huge part of their marketing. Google is not perfect. They are staggeringly weak in the face of real competition (witness Facebook vs. G+ and Apple vs. Android).
Your argument is a replay of the prevailing attitude in the mid nineties regarding operating systems (from the same company, surprise!). And then some college student named Torvalds pushed out the most influential operating system ever, and did it for free, with the help of the world.
Nobody and nothing is invincible, and people who say that a situation is unassailable are always, categorically wrong, given enough time.
Hi OP. Your points, as I understand them, are the following. Correct me if I'm wrong.
* MS is not a good example of a strong engineering org. Further, it is not a good example because Windows sucks.
* Google doesn't employ every smart person ever.
* Google can't actually compete. (ed note: in any field, or just against iOS and Fb?)
* My argument is like the argument that no one would supplant Windows.
* Given enough time, everything dies.
Points 2 and 3 are about Google. Let's start there. You're right that Google doesn't employ every smart person ever, but then, who said they did? :) But there are a limited number of search relevance engineers, and finding enough to keep up with Google is a monumental, and maybe insurmountable feat. If you want to compete with Google, you will need to have a serious advantage that supercedes this. That is just a fact.
Point 3 is about competition. I'll grant you that G+ is no Facebook, but Android is the most widely adopted mobile OS on the planet, and by a huge margin -- iOS is basically not even competitive, except for the top 5% of the market. Further, a billion smartphones will be bought this year, the most of which will be Internet enabled Android devices, and most of which will be bought in developing countries by people coming on the Internet for the first time. You tell me who's forward thinking there, because when that market comes online, it will be huge. The fact that you mention this as being non-competitive indicates to me that you might not know what you're talking about. :(
Point 1 is about the MS org. What can I say, OP, I work here, so maybe I'm not the best person to have this discussion with. But FWIW I chose this place over some much sexier jobs because the team I work with is arguably the best of its type in the world. There are bad neighborhoods, but the disparity between a good team in MS and a good team at Google is basically negligible. Also I think Windows is one of the great engineering feats of CS, so ... :| (Note that I still use UNIX at home.)
Point 4 is probably the result of confusion. I don't think no one can compete with Google. Bing has 20% market share! Clearly we can. But I do think it will be hard to compete with Google on search quality. I don't see how you can argue that.
And point 5 is obviously true but not relevant.
EDIT: Actually I now see your point 1 as saying "MS couldn't pull this off because they're not a good engineering org, but someone else could". Maybe someone else can build a better search engine, but I think what MS has pulled off with Bing is a monumental feat.
For starters, we built the entire Bing stack from scratch. No OSS. No common platforms like the JVM. Nothing like that. We started from nothing, and invented the server infrastructure, the data pipeline, the runtime that would support the site, the ML tools, everything. The fact that the site runs at all is a small miracle, but the site does not "just" run: the most remarkable thing by far is that the quality of our tooling is quite incredible, generally an order of magnitude better than the OSS equivalents. For example, the largest deployment of an OSS NoSQL datastore seems to be a few thousand nodes. The small NoSQL cluster backing our MapReduce implementation is stably deployed on a cluster an order of magnitude larger than this. This is something you only really see at companies like Amazon, Google, or MS.
I understand that the consumer market is not something MS is strong at, but I am hoping this gives you a taste of the scale and quality of what's happening behind the scenes. Happy to talk more about this if you drop me a line or skype me at `mrclemmer` :)
> For starters, we built the entire Bing stack from scratch. No OSS. No common platforms like the JVM. Nothing like that. We started from nothing, and invented the server infrastructure, the data pipeline, the runtime that would support the site, the ML tools, everything. The fact that the site runs at all is a small miracle, but the site does not "just" run: the most remarkable thing by far is that the quality of our tooling is quite incredible, generally an order of magnitude better than the OSS equivalents. For example, the largest deployment of an OSS NoSQL datastore seems to be a few thousand nodes. The small NoSQL cluster backing our MapReduce implementation is stably deployed on a cluster an order of magnitude larger than this. This is something you only really see at companies like Amazon, Google, or MS.
You lost me here. Not building on OSS seems like setting yourself up or failure from the start, particularly when you are fighting a manpower war, which is where OSS is beating every proprietary entity. OSS already powers Google and Amazon and OSS db's will scale to billions of nodes, not a few thousand.
For starters, we built the entire Bing stack from scratch. No OSS. No common platforms like the JVM. Nothing like that. We started from nothing, and invented the server infrastructure, the data pipeline, the runtime that would support the site, the ML tools, everything.
Wasn't much of the Bing stack built off Powerset, whose core technology was licensed from Xerox PARC?
For the time I was working there I'm quite sure most of the tools were internally made by MS:
- they had their own map reduce
- they had their own service deployment and management system
- their own db and nosql
- etc...
about the search and relevance they are updated quite fast, sometimes with big code replacement. This means that even if powerset tech was used as a start (and I do think it was not, it was possibly adapted and integrated -- i believe the base code was msn/live) it is now changed in most of its pieces.
>Windows is one of the great engineering feats of CS
Do I have to point out all the lame exploits and bugs which take so long to get fixed, the terrible design, the "things which should have been there years ago but we still don't have" like a decent file manager, task manager, copy utility? And AFAIK Windows made few major contribution to the theory of OSs (semaphores, threads, paging, scheduling and so on) so there's really nothing to be amazed at.
I'll spare you my opinions on Bing. Reinventing the wheel is not worth describing, no matter how beautiful that wheel is, although I'm happy for you to be a part of the team making that wheel. If only Bing had more ambition than just being a clone of Google Search, some people under 70 would actually consider migrating. But if you're happy with your default-search-engine-bundled-with-IE market share at 20%, good for you (eh, it does bring a lot of ad money). You may not call the shots at MS, but you can at least admit all the shortcomings.
You could point out all the lame exploits and bugs, but only if you want to spark a long argument that you will end up losing. There are a variety of things I don't like about Windows, and I avoid using it. But the idea that it's somehow less secure than other operating systems is for the most part a Linux advocacy myth. The fundamental security architecture of WinAPI is just not that different from that of Linux or OS X (which also has a legacy compat issue that complicates its security).
Fair enough, you know much more than me in that domain. :)
Are you going to argue about the famous general instability of Windows compared to its large competitors though? It seems like a good indicator of bad design in low level implementations.
What do you mean by famous general instability of Windows? Windows is perhaps my 3rd choice for OS, but almost all my clients use Windows and among the problems they deal with, instability is not one of them.
Hey devcpp, let me see if I have your points right.
* Windows actually sucks.
* Reinventing the wheel is not worth talking about no matter what.
* Bing sucks.
* I should admit the shortcomings of MS.
Regarding the last point, I'm kind of shocked that you think I'm a shill. I feel like I've been pretty honest about my feelings.
re: Windows sucks, that's sort of OT, but if you want to have the discussion, drop me a line. clemmer.alexander@gmail.com
re: Bing sucks, I don't see what your point about market share or unoriginality is, I already conceded that we have a lot of work to do with search relevance. I'm sort of annoyed about the negativity of your post, as from my perspective I've been pretty candid about what I think our strengths and weaknesses are. :(
re: reinventing the wheel, I don't think this is going to be something that we agree on. The way in which engineers here pull together and simply build what needs to be built is nothing short of breathtaking. I don't see how building, e.g., Cosmos shouldn't be considered an accomplishment. Should Amazon's Dynamo? What about Yahoo's Hadoop? What about anything in OSS, for that matter? I think you're not being fair here.
Not the OP - but I think you miss the relevance of point 5. Tech is littered with the irrelevant remnants and skeletons of "giant unstoppable forces". Look at Sun - in the 90s they were unbeatable, by 2004 they were quickly growing irrelevant. Oracle seems to be on the way out. Cisco too - they haven't done much exciting in a while, in a lot of circles they are viewed as the company holding networking back.
Other companies no one could unseat:
* DEC
* Xerox
* Apple (a couple of times)
* Corel
* Lotus
* IBM (a couple times in a couple fields)
* And on and on.
The point is that given a bit of time, Google will mess up, someone will come up with some new tech, and/or google will implode under it's own crushing weight.
I'm responding to the fact that OP claims MS is not a good engineering firm. I don't think that point is arguable. We are not just good, we are among the very best in the world in terms of engineering achievements.
Of course, whether we survive is another question entirely! I did not speculate on this, nor would I. Who knows what the future holds, we've just this year basically bet the company on some fairly risky things.
That said, I'm all for hating on large corporations, but the idea that Cisco and Oracle -- literally the market leaders in their respective domains -- are "on their way out" because they don't innovate fast enough is not a very convincing argument. :( Precisely who poses them an existential threat at this point? I see no one at all.
The difference between building an operating system and building a web scale search engine is that the former is mostly just work and there is a lot of material to learn from in order to inform that work. Sure, it requires talent, skill and perseverance, but there is a shitload of operating systems you can study. It also doesn't require tons of hardware, nuclear-reactor-scale power and enough bandwidth to rapidly copy, and keep updated, a significant portion of the web.
Sure, just making an OS doesn't mean it takes over the world. But guess what: you have the same problem with search engines.
This. When you do build a competitive search engine and you see how much money Microsoft has put behind theirs to essentially no avail (how many times have angry shareholders tried to get them to kill it off?) you start to realize a key fact.
If you want search engine competition you have to take Google's Ad business away.
Its weird I know but Google's search ads pay $80 - $100 RPMs and other guys ads pay $30 RPMs. If Microsoft could use Google's ad network they would be a solid contributor to the bottom line of Microsoft (which is why they can't of course).
If you could magically peel that Ad network/agency into its own entity and require it to give non-discriminatory terms to everyone, I believe we would have a pretty vibrant search space. My reasoning there is that the money associated with search advertising would fall into buckets that were much more closely aligned with market share, as opposed to today where someone like Microsoft can have a large share of the search 'eyeballs' but only a fraction of the revenue because their Ads don't have the RPM numbers.
Both Yahoo & Microsoft did a really bad job for a really long time at simply providing a platform that is comfortable for advertisers to use.
I have run ad campaigns on Adwords continuously for 7+ years. Throughout that time I have run ad campaigns on Yahoo, now Microsoft Adcenter, on and off. The last I checked their ad platform was about where Adwords was in 2005.
Yahoo had some very reprehensible things on their platform. I had to shut off all of my campaigns because someone at their company was changing what my ads said without my permission. Besides being a legal issue for Yahoo at the time, it put me in a position of unlimited liability. Fortunately, I never witnessed that behavior after Microsoft took over. Yet, Microsoft's platform was just too difficult to get working.
That was probably $5 million + in missed advertising revenue from me, just a tiny advertiser. I can only imagine the billions of dollars of revenue Microsoft and Yahoo lost for failing to take seriously the search advertising marketplace.
It is not clear that agency is the issue here. The search ad business is a long-tail business. People who read and comment on HN, who understand how search works, likely are heavy users of search yet contribute little revenue. But hackers are important to Google in other ways because they are the tech trend setters who turned Google into a verb and told their computer-illiterate brethren to just "Google" it. Now just imagine what some of those less literate, who believe in palm reading or tarot cards, would think when faced with a computer that appears to half read one's mind. I mean these are the real gold mines that the ads people are looking for. And once they get comfortable they are not just going to switch based on some technical merits they've never even heard of and the ad market will continue to pay more to those who can deliver more of the "gold mine" type users.
Honestly, the problem is that microsoft built a search engine like google, which needed to reach feature parity and until parity is reached consumers have no reason to use it. If feature parity is used, consumers have no reason to switch as the cost of switching does not outweigh the benefits of it, therefore you have an unnecessary product that you have to pump a shit load of advertising on, which creates more costs and no revenue.
The correct way, I believe, is to compete on non-consumption and capture those users, therefore you start small and execute well and scale up as users grow. You avoid the comparison to Google and can sneak in the back door without the crazy upfront costs. Imagine if microsoft focused on Knowledge graph before google did? It would be very interesting times, but every google competitor just competes head on. At the end of the day its about getting users and they don't care about the slight change in your algo, only in the way they feel when using your product i.e frustration, delight, anger, surprise, etc!
The advantage of being a startup is precisely that your mission need not cohere with the investments of a large corporation. You can do what you like.
From MS's perspective (again: I work here, but my opinions my own blah blah) the problem is this. People use computers to access the Internet. MS can't just be the OS and the browser used to access the Internet to maintain its lofty position as a field leader -- if MS's job is to supply the Internet as a service to people on MS devices, then it is mission-critical that it also be the landing page of the Internet. If MS gives up Bing, it might as well give up all consumer investments IMHO.
It is (IMHO) more important that Bing exists and is mostly functional than it is that Bing is equal or better to Google in every way.
Of course it is a huge priority to make Bing a viable threat in its own right, but what I'm saying is that this is not the only consideration.
That's an amazing insight, thanks, i hope potential entrepreneurs take note of this and realize the problems corporations face with innovation. It also makes sense as to why corporations are so quick to acquire a startup but also why most times the acquisition is not successful.
I disagree that this is the main reason why MS is behind Google on search.
I think the main problem of MS is that it is copying Google search and not providing anything substantially different or better. In other words, fighting against incumbent is not about spending more money than incumbent and doing the same. It is providing service for "niches" where incumbent is not willing or not able to provide service.
Here is an example:
Google shuts down Google Code search. I'm not happy. But, I was thinking... There is Bing... They will maybe jump in and provide state of art code search. But... nothing happen.
Is not the whole story of technology doing more with fewer humans?
Google could simply stay ahead of the curve, but this assumes all of that human capital is being dedicated specifically to building the most effective search engine.
What I see now is an advertising search engine. The friends who don't believe me are the ones who haven't turned Adblock Plus off in the past 3 years.
What about targeted search engines? Google has gotten progressively worse for technical searches as it has been improving generalized "what you mean" searches for everybody else.
Personally? I think there's lots of room for innovating search. Probably not in that area, but in a lot of other areas. Consider, for example, that the majority of the world will has an Internet connection only via phone. Mobile search is an entirely different beast. It is not clear Google's solution is the best here, and because the pattern of using Google on mobile is shaky enough that it could be supplanted with a convincingly better solution. That's to say nothing of the fact that Google's success in developing countries is actually not at all a given.
The dominating cost is not hiring smart people to work on the problem, it's hiring enough smart people to work on the problem.
Consider. Here at Microsoft, we employ a search relevance staff of -- what, 20% the size of Google's? And Google's engineers are rock solid. MS might be able to make up a small margin, but our search relevance staff -- good though they are -- cannot compete with a brilliant workforce that is 5 times larger than them. If we are going to actually compete with Google, the problem is not hiring smart people, it's hiring 5 times the current number of smart people employed for this task at MS.
Never mind buying such a team. MS can afford that. How do you even find that many people in 2-3 years?
When you consider the rest of the data, the case for building something like Google really begins to look grim. For example, how much does it really cost to build a search engine? We've poured at least tens of millions of dollars into just search relevance (I'm not even counting infrastructure). That's good mileage, considering that this way is littered with the corpses of companies like Cuil, who made investments in this area and failed miserably. Still, while this has been a great deal for us, we're still not there yet, and it's not clear we will be in the very near future. And so it's worth wondering: if MS can't buy something like that, realistically, who can? DDG? lol no.
In the end, this is the true dominating cost of building a search engine: people capital. Other bottlenecks, like engineering debt, politics, etc. pale against the sheer, awe-inspiring investment of Google in people capital.