No, I can't; if the lower income households have been splitting their earnings into a greater number of households, a growth per person would be masked by a drop in total income per household.
To use figurative numbers, if a couple earns $30k each, then they split but both get a raise of $20k, the household average drops from $60k to $50k despite the large salary raises.
If that is true for 80% of the population, then that is similarly a cause for concern.
While you are correct that change in household size could be muddying the income numbers, I would say that the difference in behavior between top 20% and bottom 80% must reflect something driving the two apart.
To use figurative numbers, if a couple earns $30k each, then they split but both get a raise of $20k, the household average drops from $60k to $50k despite the large salary raises.