No need to use git, since you can just use diff(1) to find the unfollowers between any two dates. I've been doing this for a while with a Go program I created[0].
Unfortunately, people who change their Twitter handles (which I found surprisingly common) show up as false positives - you either need to filter these out manually or match against the user id[1].