Computers don't care about mathematical equivalence. For instance, dividing by a number is generally slower than multiplying with the inverse of that number. Also, memory access patterns can make a difference of night and day.
I'm not sure though where exactly the idea that GA is slower comes from. It's all down to the application and implementation. Perhaps it's because GA generalizes so well to higher dimensions that many libraries are overly generalized and thus slower than vector algebra libraries that need special cases anyway.
The 3D case is often handled using a conformal projective geometric algebra, which is 5D and requires 2^5 = 32 elements for a general multivector. This is twice that of a 4x4 matrix, and and eight times the size of a 4-element homogeneous vector as typically used in most 3D engines.
Of course, there are all sorts of optimisations. Most GA libraries are actually based on code-generation and support various "subsets" of the full GA to efficiently represent things like vectors only. From what I've seen, even a well-tuned library has approximately a 25% overhead for hot paths and 50-100% is typical.
This is a bit like representing a simple rotation like the hand of a clock, with either just one angle, or a two-element unit vector pointing in the direction of the rotation. The one-element angle has a bunch of special cases, like have to be checked to see if it goes past 360 degrees and then reset back to the 0..360 range. Representing this with two numbers just requires multiplication with a matrix, which involves no conditionals or modulo arithmetic.
This is analogous to the vector vs geometric algebra. Typically, GA has no special cases and "just works", but it does so by "uncompressing" the compact vector representation. It's going to be slower.
I'm not sure though where exactly the idea that GA is slower comes from. It's all down to the application and implementation. Perhaps it's because GA generalizes so well to higher dimensions that many libraries are overly generalized and thus slower than vector algebra libraries that need special cases anyway.