The lack of location tracking looks like the most significant difference to me. While the git-lfs documentation does mention that different git remotes can have different LFS endpoints configured, all git-lfs knows about a file is its SHA256. So how can it tell which remote to download the file from? The best it could do is try different remotes until it finds one that has the file.
I hesitate to say this means git-lfs is not distributed at all, but it seems significantly less distributed than git-annex, which can keep track of files that might be in Glacier, or on an offline drive, or a repo cloned on a nearby computer, and so can be used in a more peer-to-peer fashion when storing and retrieving the large files.
That doesn't work very well. Consider what happens if two different clones of a repo update the metadata last-known url at the same time with different urls. Merge conflicts.
This is why git-annex uses a separate branch for location tracking information, which it can merge in a conflict-free manner.
I hesitate to say this means git-lfs is not distributed at all, but it seems significantly less distributed than git-annex, which can keep track of files that might be in Glacier, or on an offline drive, or a repo cloned on a nearby computer, and so can be used in a more peer-to-peer fashion when storing and retrieving the large files.