Andro Linklater in Measuring America says this is one of the great tragedies of forcibly relocating Native Americans from their traditional lands: the loss of the oral histories bound to routes and landmarks on those lands. So the theory isn't without some precedents. I'm not sure what evidence he puts forward for this though.