The tracks between Toronto and Montreal are owned by CN and freight has priority over VIA. VIA trains are often delayed having to wait for freight trains to clear and can only be scheduled for times and speed that do not interfere with freight.
It takes 3 days to travel the 4500 km of rail from Toronto to Vancouver because the first 1500 km are through muskeg and around lakes where building straight, flat, protected 200 km/h high-speed tracks is just not an attainable option. Even if you managed 200 km/h the entire way, it's still 2 days travelling.
Planes travel about 800 km/h and can go in a relatively straight line without fear of a moose or a washout. It's 5 hours Toronto to Vancouver and 4.5 hours the other way because of tail winds.
I used to somewhat regularly take Toronto <-> Kingston, and I don't believe I was ever delayed by freight on that leg. Maybe the CN ownership prevents them from doing other kinds of upgrades on them, but they seem to get pretty decent priority during the day, particularly compared with the Toronto -> Vancouver route where you can sit on a siding for 4 hours. And it's not even really a matter of priority; the issue is that the siding is only long enough to accommodate the passenger train— the freight train going the other way is way, way longer.
In any case, I don't think anyone has ever seriously suggested HSR for the train out west. VIA has simply embraced that it will always be slow and impractical, and they market and price it accordingly— it's supposed to be an experience, like going on a cruise, not a practical means of getting to a destination.
It takes 3 days to travel the 4500 km of rail from Toronto to Vancouver because the first 1500 km are through muskeg and around lakes where building straight, flat, protected 200 km/h high-speed tracks is just not an attainable option. Even if you managed 200 km/h the entire way, it's still 2 days travelling.
Planes travel about 800 km/h and can go in a relatively straight line without fear of a moose or a washout. It's 5 hours Toronto to Vancouver and 4.5 hours the other way because of tail winds.