The answers to this are on a person by person basis, but I know it doesn't have anything to do with middle class poverty or a single man living like an "average" family of four.
Here is what I think connecting with others means: I connect with someone when, in my heart and mind, I understand their needs and feelings.
If my everyday experience is different from a person's daily experience, my ability to connect with them decreases radically. For example, I have great difficulty connecting with people who do not read.
So, as I imagined spending 10x what I was spending before on myself, I realized that I would lose my ability to connect with a huge number of human beings. Hence, the vow.
Your definition of connecting with others would be close to mine as well, however, I think I could accomplish those things no matter what my everyday experience was or how much money I had in the bank.