This feels a little like entitlement. Like the candidate is saying "You owe me an explanation". They already gave you a big chunk of their time. You gave them a chunk of yours. Nobody owes anyone anything.
If you want to speak in such terms, that would be the case if the prospective employer visited the prospective employee for the interview. When (99.9999% of the time) it's the opposite, I can say that you could give them some feedback to appreciate their...commuting, let's say?
