The layers are relevant if one of those layers checks that the request is valid and legal. Assuming it's true that email address in, [access check layer], data out, then we have nothing to worry about.
Well, we still have to worry that only the NSA has access to the layer. But even with ongoing crypto breaks I trust computers more than I trust people (in a world of spear phishing) to do the right thing.
Being worried about unauthorized access to your server is not a new worry and is not limited to this topic. So if that is our only worry, then this whole thing is a non-issue.