What is the best cheap setup for initial development? Low cost, easy to change to different languages (or web server) if first choice doesn't work, easy to move to a colocation site if that time ever comes and easy to administer. GUI isn't required.
My last webserver was circa 2000 with Apache and Redhat.
For initial development, you probably don't even need a server. There's pretty much always an easy way to have a local server run on your computer for development.
When you are ready for a server (probably not until you're about ready to go live), I'd suggest Slicehost. They're well-documented and cheap ($20/mo. for their smallest server!), and you can scale the server all the way to 15.5GB of RAM in just a couple of minutes.
You could look at Linode too though. It seems like you get a little more RAM for your buck, especially on the smaller end, but you can only go to ~2.8 GB of RAM on one server.
As for your development computer, I love Ubuntu, but it really depends what you want. If you want to do everything through an IDE, it depends on your language, but the best IDE may be for Windows. If you want to do much of anything through the command line or think you might in the future, develop on either Mac or Linux. Their shells are just way better. There is always Cygwin, which sorta fixes the problem, but I think true Unix works better.