I am currently playing with the idea of moving an application from all-GAE to part GAE, part EC2 with some spot instances running jobs when cheap enough. According to my sloppy math, this should reduce the load on the GAE side by at least 60%.
Anyway, this is all theoretical (in the worst sense of the word) - the app is not even public.
But is it really worth the headache of such a Frankenstein architecture?
I initially fell in love with the "no sysadmin" aspect of App Engine, and started building apps around it. Eventually I realized that (for me, anyway) the upside isn't really worth the trouble of having to contort my apps to work in Google's sandbox- can't run SQL, have to deal with datastore timeouts, CPU timeouts, etc. When you're done coding work-arounds for all of these things, are you really coming out ahead?
This is so true. The first step to correcting a problem is admitting you have one, and everyone on App Engine clearly does (locked-in). What I'm doing about it? Moving to Tornado/MongoDB/EC2 as quickly as possible.
That is my current project's design. Web frontend stuff in GAE and AWS doing the backend processing. AWS is needed to run media servers which can't run in GAE. I was going with it despite the disadvantage of the split architecture because I liked GAE. Now with the new pricing model kicking in, the frontend cost has skyrocketed and it doesn't make sense to run frontend stuff in GAE.
Also the way Google handling this confirms my fear of proprietary lock-in, leaving a really bad taste in mouth. I meant I've spent lots of time to develop in their framework to work around their quirks and limitation. They have been advocating to developers to optimize for CPU time and that's what we did. Now they found out they don't make enough money with CPU optimization and change the whole billing scheme, screwing all the efforts done by developers. What would stop them to change again tomorrow if they find instance-hour is not making enough money for them?
I went through this thought process. I really liked Python and auto-scaling, so GAE seemed really appealing. But the combination of lack of SQL and inflexibility of architecture led me to want to create the sort of combination architectures you describe with EC2 instances or whatever.
Ultimately, after a lengthy debate with a friend, I concluded that Heroku was a sufficiently better option that it was worth learning Ruby (which I've been writing more or less as an uglier Python anyway).
Anyway, this is all theoretical (in the worst sense of the word) - the app is not even public.