As the proud parent of a three year old: don't feel guilty about holidays¹ for two if your kid has a safe and pleasant place to stay. Staying with their grandparents (a common choice) or other trusted guardian helps them experience a different household, and if you start doing this young it is never a big deal.
1: Preferably with lots of intimacy. A good part of our luggage for getaways is sex toys.
1: Preferably with lots of intimacy. A good part of our luggage for getaways is sex toys.