And if you move away (to escape your past or for other reasons), any new village you move to will treat you as a tolerated outsider for 20+ years because you didn't go to the same kindergarten as them.
And, by and large, they were right to do so. In the absence of a very strong central government, the only way to protect yourself from criminals (or other social problems) is to assume all outsiders are problematic.
Allowing "escape your past" is generally not socially desirable.
