I have been using Piwik/Matomo for years, and have never experienced this.
That being said, ultimately it's a frontend on a MySQL database, so there's lots of ways it could theoretically be slow -- MySQL isn't configured/tuned properly, the database server isn't resourced appropriately for the amount of data it's hosting, etc. But this is going to be the case for any self-hosted solution.
Although it was about five years ago, yes, my potato server (still running the same one today) couldn't handle that by a long shot. Same with Wordpress, Owncloud, and other web-based things that became a little oversized.
Yes, I worded that poorly. I meant loading Piwik's UI; it was cripplingly slow and often timed out. I had little luck in narrowing down the cause, but it was hardly a thorough investigation on my part, IIRC.
Assuming that your disks, DB, CPUs and RAM were all ok, maybe you did not set up PiWik's jobs that are supposed to aggregate the metrics every XX hours/days... ?
