I think it's because a lot of people saying that don't really know (and I mean really well) their scales; if you do then it's "obvious" that (say) an F is a semitone above an E, despite that gap on the stave looking the same as F to G. All the musicians I know who can read well know their scales inside out. I'm a bad reader, but my reading has improved considerably over the last couple of years as the scales have become something that I know really well (in simpler keys). Then it turns into a "how could you -not- know that?" kind of deal, internally. I think that's the missing piece from most people's ability to read well; not the actual reading, but grokking the key that you're in so it just becomes 'obvious'.
I'm not there yet!