I work from home, family of five and 100 is not enough.
all my tv comes over internet. three TVs, often two on at the same time. add kids with tablets, and streaming radio, teleconference and virtual desktop and 100Mbps is struggling.
Latency is the tough part, less latency matters for most interactive activities.
i work from home: 360Mbps up, 35Mbps down.
I think it's fair to say from your description that your bandwidth demands are greater than average. (Also I think you swapped your up and down, unless you've got a really strange connection.)
I have the similar usage to you. Cord cutter, four heavy smartphone/tablet users, remote desktop, etc. About a year ago a badly done patch-panel punch on my part slowed down our internet from gigabit to fast ethernet speeds. (Bad connection on one of the four pairs.) Nobody noticed.
I've got 3 gigs symmetric at my house, with 10-gige in the wall, and Ruckus APs that can do at 500+ mbps if using Apple products. I can't think of a way to use even a fraction of the bandwidth.