Second on the idea that it should be quick and easy to make useful stuff, but I have to disagree on PHP. Most of the example code out there is terrible, out of date or both. It's very easy to make a mess in PHP.
Python seems to be the teaching language of choice these days, and it lets you do useful stuff without getting in the way. It's certainly not the only language to learn, but I'm inclined to say it should be the first.
There's nothing wrong with PHP 5. As long as he's learning about all the latest stuff, it's a perfect language. It features functional aspects to get him started and then he can graduate to OOP concepts. Since he's learning from scratch, stuff like objects and public/protected/private members might be a little farther down the line. PHP lets you get the basics of variables, arrays, functions, etc. out of the way without having to switch languages or set up some construct you have to ignore until later, when you're more adept at the language.
Yeah, PHP isn't an excellent language but it does do one thing and do it well. Just 99% of examples or scripts you find are absolutley horrible. Seriously, I've seen blog posts on PHP from IBM which contain bad programming practices.
I'm not sure that this "please think of the children" argument -- the one which asserts that the sight of PHP will forever warp the minds of children and risk turning them into script kiddies, or Blub programmers, or crack addicts -- can be nuked. It's really old. People were apparently using it when I was thirteen -- except that the guilty party back then was not PHP, which was more than a decade in the future, but BASIC. And I obviously didn't know about the controversy at the time, because I was too busy using hunt-and-peck typing to rekey my BASIC apps over and over again. (At the time my school's brand-new Commodore 64s didn't have tape drives, let alone floppies.)
And the kids were alright back then, and they are alright now. Applesoft BASIC makes PHP 5 look like Haskell (we're talking about a language with nothing but global scope, here), but the generation that built the Web grew up using it and it didn't hurt anyone. The smart kids just moved on to better things as they became aware of them.
Teach the kid something fun. For a thirteen year old, that's very likely to be Javascript, Actionscript (i.e. Flash), or PHP, though I certainly might give Shoes or Hackety Hack a try, or maybe this Scratch thing: http://scratch.mit.edu. Try several of these and see if any of them stick. But don't get hung up on the details. The guy is thirteen. There will be plenty of time for him to learn how ugly, fragile, insecure, opaque, and unmaintainable his code is. Try to let him have some fun and get hooked before he's forced to learn the truth. He won't sit still for it, otherwise.
As an occasional reader of The Daily WTF, I don't think it's fair to say that this sort of thing never hurt anyone.
Every language that has been around for a while has its own culture and traditions. The culture of PHP is a better reason to avoid it than the language itself. I spent a lot of time un-learning habits I picked up from languages like PHP and BASIC. I suspect I would know a lot more now if I had started with Smalltalk[0] and not had to waste as much time un-learning bad habits.
Python seems to be the teaching language of choice these days, and it lets you do useful stuff without getting in the way. It's certainly not the only language to learn, but I'm inclined to say it should be the first.