> if someone wants your keys, they can break your knees
This doesn't seem relevant here. The government can make all the secret attempts to read your communications remotely that they want. If they want to take the strategy of torturing you until you hand the information over yourself, at a bare minimum they need to be willing to admit to doing it. It's not a threat to worry about, from the US government.
I think that if a government agency thinks it can get away with away with wiretapping on the PRISM scale, it or another government agency also will think it can get away with abductions, torture, and killing.
The US government is known to kidnap people [1], torture people [2], and kill people [3]. It wouldn't be that surprising to learn that some people were victims of all three.
> It's not a threat to worry about, from the US government.
You are first of all assuming the worries are coming from US citizens. People outside the US can obviously worry about the US government or its allies breaking their knees, it is a widely documented fact of life in many US war zones. We know that the US has officially been torturing people it suspected of terrorism for information (Guantanamo, CIA prisons in Eastern Europe) .
Second of all, we also know from the past that the FBI has been involved in campaigns of infiltration, blackmail, threats, incitement to violence, and almost certainly assassination of civil rights organizations and leaders (COINTELPRO). This was investigated at the time by Congress and new laws were put in place to prevent this type of behavior, but we have no guarantee that it didn't resurface in some form, especially in the current political climate (especially by not only the 'war on terrorism' started by Bush and escalated by Obama, and enthusiastically supported by Trump).
Overall, I think it's perfectly reasonable to fear the US government may physically force you to give up secrets, obviously so if you are not a citizen, and quite likely even if you are one.
Realistically speaking, if your scenario is that the US government might capture and torture you for encryption keys, then your number 1 priority should be physical security of your communication devices and the people who operate them. Your number 2 priority should be preventing other side channel attacks, e.g. the operating systems you're running on your endpoints and things like the Management Engine on Intel chips and the equivalent on AMD chips, as well as other possible backdoors in the hardware and your supply chain. Remember, the NSA intercepts mail-order hardware and modifies it and the CIA runs hardware companies.
Once you have taken care of these priorities, you can start worrying about the soundness of your encryption. Inventing safe encryption if you're not overly concerned about performance is really not hard, even experienced laymen can do that by using existing cryptographic primitives. You can even make it quantum safe. (It should be, in the described scenario.) If you think that is not within your capabilities, then you're probably right, but then you've already failed at task 1 and 2 anyway.
For the remaining 99.9999% of the population this is not a realistic threat scenario, and it's best to use a well-established cryptographic library with the recommended defaults.
This kind of thinking is what inspired by (root) parent comment. If we don't like our government putting people in the "capture and torture" risk management scenario, we need to act politically to prevent that from happening, because there is very little we can do technically to prevent such a thing.
This doesn't seem relevant here. The government can make all the secret attempts to read your communications remotely that they want. If they want to take the strategy of torturing you until you hand the information over yourself, at a bare minimum they need to be willing to admit to doing it. It's not a threat to worry about, from the US government.