Ah, I had neglected the effects of interstellar plasma.
I assumed the magsail operated in a similar fashion to a solar sail, depending on the solar wind. If the destination's solar wind was sufficient to slow us down, I expected that the Sun's solar wind would be sufficient to speed us up.
If you launch a strong magnet in interstellar space, it will slow down relatively to plasma by deflecting charged particles.
In order to speed up, you need to spend energy.
The symmetry is broken between accelerating and slowing down relatively to interstellar plasma (Edit: or solar wind).