If you suffer a power loss and/or MongoDB exits uncleanly then you will need to repair your database.... it involves MongoDB going through every single document and rebuilding it. When you have databases at the size of ours, this would take hours.
Yes, and if you run a cluster long enough you WILL suffer unplanned power loss. Usually due to human error of one sort or another. It's basically impossible to eliminate entirely.
- that seems really scary