Well, we still have to worry that only the NSA has access to the layer. But even with ongoing crypto breaks I trust computers more than I trust people (in a world of spear phishing) to do the right thing.
Being worried about unauthorized access to your server is not a new worry and is not limited to this topic. So if that is our only worry, then this whole thing is a non-issue.
