I was doing 1.5 million Ray's per second against scenes with 100,000 polygons back then on an AMD64.
Acceleration structures are everything. Traversing them quickly and having dynamic updates is hard.