Seems like your issues were not with WP itself, but with whatever plugins and themes were added to it. Avoiding WP entirely for this is like avoiding a programming language because the 1 developer you had experience with sucked at it. WP itself can be very fast, as is evident by a ton of high profile sites running it (CSS-Tricks, TechCrunch, New York Times, Time Magazine, etc). I'm not a fan of WP myself, but that's just because I don't like how its built and how it entirely avoids modern programming standards, not because it is slow, which it most definitely doesn't have to be.
