Surely IRS are civil service and so are required specifically not to worry about voters. If you're a non-voter you still get "service" from the IRS (though many might wish they didn't!).
You can vote out your representatives but you can't usually directly vote out any element of your civil service.
You can vote out your representatives but you can't usually directly vote out any element of your civil service.