I grew up in a super "dirty" household. Shoes in the house, dog, no house cleaners, the whole family did our own gardening and lawn care, etc. No allergies. In fact allergies were practically unheard of in that place and time.
Yeah I don't think allergies depend on external factors that much. My mother and I have pretty equivalent childhoods even down to same locations and she's wrecked with allergies and I have none, yet my health is worse overall (despite me having a healthier lifestyle).
While I believe some exposure helps build resistance ultimately your overall health will be influenced by stuff beyond your control AND your lifestyle/activities combined.