The problem with high-speed rail is that for decades it will only benefit large urban population centers. As such, people in rural communities or more sparsely populated cities have little to no incentive to go along with funding it. If a person has to drive for an hour in the wrong direction just to get on the train, then the train has no real benefit to them.
Who says the train has to benefit everyone? The government spends plenty of money on things that don't benefit me. Should I be demanding a refund on my taxes for those things?
Yes, but the census defines an urban area as containing over 50,000 people. There are going to be an awful lot of urban areas that are not at all conveniently close to a high speed rail station.
If you count by census statistical area, the 10 biggest MSAs contain ~80 million people. There's 180 million people in the top 50 metropolitan areas, well over half the USA, all living in conurbs of at least a million people.
At least in the case of the California plan, if you have to drive an hour to get to one of the stations then you live in BFE and would have to drive an hour to get any damn place. It's going to have a station in Hanford (pop. 50k)!
I don't know about that. California's high speed rail will go through Central Valley, which is the poorest part of the state. And for some people, it might be driving an hour in the right direction to take the train.