> But the weird thing is that if you take two adults and compare their IQs the difference there has a negligible weighting for environmental factors. Modern studies put the heritability of adult IQ upwards of 80%.
To be clear heritability is about the differences within individuals and not a raw value by itself. For instance if we have a trait with 80% heritability and one individual is measured at 120 against the average of 100, then this would mean that 4 points (20% of the 20 point difference) would be attributable to non-genetic factors. Some confuse the term to mean that e.g. 20% of the entire value would be attributable to environmental factors. In other words a heritability of 80% does not mean that 80% of a trait is genetic - it means that 80% of observed differences in a trait can be attributed to genetics.
As a not directly related aside this also leads to another really interesting aspect of heritability. It changes as the average environmental situation changes. As we get closer and closer to environmentally equal, any given trait's heritability approaches 100% as the only differences left between individuals become genetic. As an obvious example consider height. Malnourishment tends to stunt an individual's growth. So in an area where some significant chunks of the population were malnourished and others were not, you'd see a much lower heritability for height than in places like the US where that environmental factor has been mostly eliminated.
20% isn't negligible.