As Rob mentions in his post, the more dotfiles there are in $HOME the slower path resolution for any subfiles becomes. How do we navigate to ./src? We open the directory and read all the entries until we find the one called "src". What happens if we encounter a morass of dotfiles beforehand? src takes a while to find. The clutter may be invisible to you, but it does gum up the works.
For what it's worth, most modern file systems (JFS, XFS, ext4, reiserfs, btrfs, ...) have logarithmic (or better) directory lookup times. This is achieved using hashes and b-trees (or even hash tables).
