Raw php also has to initialize connections, and it does really well, so I don't think that's it. To me, that points to overhead from initializing the objects as the bottleneck. I would have thought that with APC, this wouldn't be a major issue, though, so I wonder if that's still not it.
APC only caches the opcodes, so the interpreter doesn't have to parse your code. But the framework still has to set up itself for each request individually. Parse configuration, create objects, etc. It adds up quickly.
