Yes, that's true for most domains. All the other servers have to cache responses from those two servers. (For the record, the two yp.to servers are currently djb's machines dancer and forcewall.)
What would happen if he suddenly decided to unplug dancer and forcewall forever, would all the other DNS servers in the world fail to resolve yp.to after their cached responses expire?
http://cr.yp.to/djbdns/intro-dns.html
under the "Multiple Servers" section, he says
"To protect against computer failure, there are actually several root servers, several .to servers, and two yp.to servers."
I don't understand, is he saying there are only two DNS servers in the world that you can contact to resolve yp.to?