When Ruby or Python is too slow you can switch to something else. When JavaScript is too slow you have to either pay someone to make your code faster or pay someone to make the browser faster.
It mostly did before V8 was created. A few people ran JS outside the browser using Mozilla's Rhino, but it wasn't very common. Nodejs probably was the first widely adopted platform for non-browser JS, and it was built on V8 after V8 already existed. So I believe historically server-side JS did not drive JS performance.
It should be mentioned that Microsoft has had server side Javascript (well, Jscript) running on IIS since 1997. I used to write data driven websites in it.
