You're right -- the optics are everything. The flashlight fetishists love round symmetrical beams, which is exactly what you don't want in a headlight.
You want to focus almost all of the light at the horizon, with some spill below and a sharp cutoff above. Anything else is wasted. See: http://peterwhitecycles.com/plight.asp
I agree. It's just hard to find good headlight-worthy optics for DIY LED or Halogen lights. You always have to buy the whole package, always for $too.much from someone like Peter White, or the local bike shop.
Even my "good" headlight (15W, ~300 lumens by my guess) is just a stupid 6V MR11 bulb with a clear waterproof lens in a waterproof housing. No optics. It's just a farking accent light that can be bolted to a bike. Über lame.
You want to focus almost all of the light at the horizon, with some spill below and a sharp cutoff above. Anything else is wasted. See: http://peterwhitecycles.com/plight.asp