I'd say that programming falls into two categories. I don't quite know how to describe them, so here are two examples:
1) Your average database/object driven application. There is user input. This translates pretty directly to output. There may be some interesting algorithms in between, but it's pretty much UI programming.
2) Solving difficult problems. A logistics routing application. The bulk of the code is based on graph theory, reducing the problem perhaps into smaller segments that can be solved in different ways. The UI is just the surface.
There is a need for both types of programmers. The first type are increasingly graphic designers who decided to learn PHP or some similar situation. There are tons of great applications out there that really aren't that complex (except when scaled to extreme proportions - ie Twitter). Wordpress, Gmail, even Microsoft Office isn't terribly complex (at least, the 90% of it that everyone uses).
The second group seems to be a bit more selective. Applications like Dropbox, which need to route data in the most effective way, or Adsense, in which profit needs to be optimized in choosing what ad to display as a page loads. These are complex problems, with complex solutions. That's where O(n) analysis is important. That's where an understanding of graph theory and established algorithms is critical.
While these two groups aren't completely exclusive, I'd say many self taught programmers find themselves in the first group.
I think you've actually got no clue how complex and hard the average database/object driven application can become. Nor do you seem to have any idea what is actually entailed in a large CRUD application.
You claim that GMail isn't complex. Jeezus, a tiny % of programmers are capable of recreating that well. Or Microsoft Office! Go get the source code to OpenOffice and say that again. Think about the problem a bit harder and you'll realise it's a freaking compiler that's far more complex than most programming languages.
Far, far, far, far more complex than Dropbox.
I bet the hard bit of Dropox was the seamless UI integration with the OSes and not the routing of data. The data routing was probably solved in an afternoon. That was probably the fun bit.
You seem to have no clue about what is actually hard in programming. Hard problems, pffft.
It's as if I went to projecteuler.net and looked at some code and went 'oh, look, implementing algorithms is easy. Any A-level maths student could do that...'
Graphic designers generally aren't capable of programming or maintaining a large CRUD app, and when they try what usually happens is that some professional has to come and gradually rewrite the whole thing. Without breaking anything. That's hard.
If they are capable then they usually call themselves a programmer, and rightly so. And if they wanted to solve the 'hard' problems, they could. They'd just have to learn about a different area of computing. And I'm mightly jealous of them for having mastered two hard subjects instead of just one.
So next time you want to declare that 90% of programmers could be replaced by people with no programming skills, go look at some source code and then shut up.
Sorry. Those examples are admittedly more complex than I made them out to be. I mentioned scaling to be a more difficult problem. There are some master architects in all those projects, who deal with hard problems, but a large code base is not the same as a complex problem. Large pieces of software are many small and simple problems held together by a complex glue. The architects deal with the glue, but category one programmers can so most of the other stuff without too much difficulty.
No, there generally aren't master architects. I'm surprised to hear you talking about master architects, in traditional programming circles the term is usually met with scorn. There's a perception that people who call themselves architects or talk about architecture tend to suck at programming. That they like talking about code, but in general are not actually capable of doing it.
Hearing someone say they're a software architect or that a company has a software architect is usually a big red flag to run like hell.
Also a large code base is identical to a complex problem. Check the definition of complexity. A complex problem can be broken down into simple components.
Finally the best glue is not complex, it's simple. Complex glue tends to result in brittle magic, a poor abstraction, like traditional asp.net compared to something far more elegant and smaller like ruby on rails.
I don't think "average database/object driven applications" constitute the majority of programming, and I strongly suspect that Microsoft Office is not as trivial as you say :-) See Joel's article on the "five worlds of programming": http://www.joelonsoftware.com/articles/FiveWorlds.html Each "world" has intricacies that people from other worlds aren't aware of, and there are likely more worlds than just these five.
Actually, the second category should be differentiated further:
2.1) Solving information problems. What you describe.
2.2) Solving physical/mathematical problems. Think signal processing. Think physical simulation. The bulk of the code is based on some mathematical theorem or linear algebra that is just too complex to be calculated by hand. The UI is just the surface.
The 2.2 group seems a bit more selective. Applications like audio workstations, simulation grids or wheather forecast centers need this. Universities use this for research.
While these two groups aren't completely exclusive, I'd say many CS graduates lack the engineering/mathematics/physics prowess needed for the problem spaces encountered in 2.2.
1) Your average database/object driven application. There is user input. This translates pretty directly to output. There may be some interesting algorithms in between, but it's pretty much UI programming.
2) Solving difficult problems. A logistics routing application. The bulk of the code is based on graph theory, reducing the problem perhaps into smaller segments that can be solved in different ways. The UI is just the surface.
There is a need for both types of programmers. The first type are increasingly graphic designers who decided to learn PHP or some similar situation. There are tons of great applications out there that really aren't that complex (except when scaled to extreme proportions - ie Twitter). Wordpress, Gmail, even Microsoft Office isn't terribly complex (at least, the 90% of it that everyone uses).
The second group seems to be a bit more selective. Applications like Dropbox, which need to route data in the most effective way, or Adsense, in which profit needs to be optimized in choosing what ad to display as a page loads. These are complex problems, with complex solutions. That's where O(n) analysis is important. That's where an understanding of graph theory and established algorithms is critical.
While these two groups aren't completely exclusive, I'd say many self taught programmers find themselves in the first group.