That's a similar but different anecdote. The one I am thinking of he specifically is talking to someone's wife while she is crocheting. It may not have been Feynman though, I CTRL+F'ed "Surely You're Joking" and wasn't able to find it or I would have cited my sources in the parent comment.
