They can (and often do, at least for deportations from the UK) demand to see evidence that the deportee is, in fact, one of their citizens. Which can be difficult to prove without a passport. We have people who have been in immigration detention for years because they won't cooperate in getting an emergency travel document, or their 'home' country doesn't accept that they have a right of admission there.
Your home country can't refuse you entry, regardless of identification. Deportees don't get to stay in the US just because they lost their passport.