That letter is written by lawyers–trying to explain the specifics of "test cases" versus "source code" is unlikely to be a useful use of space. (Plus, if you can get the entire project taken down, why not go for that?)
Do you know of anything that might be applicable here for arguing a distinction between test cases and actual source code? It's a silly example but bringing a replica weapon through airport security is legal, I imagine, if done to test the security system. From a legal layman's perspective, testing on real data doesn't show intent but is instead good engineering to show that the code is verified for correctness.
The point being made in this takedown is that youtube-dl can be used for copyright infringement and is being used for copyright infringement, and the test cases pretty much do exactly that. (In fact, if you're a lawyer with a bit of an idea of software engineering, you might make the argument that the code is testing the correctness of whether it can infringe copyright!)
As an (hastily made, non-expert) example consider a particular gun called the "PandaHunter 9000". Now, pandas being endangered species and such, and having reports that people use buy the "PandaHunter 9000" to kill pandas, you might consider banning the purchase of such a gun. Of course, the manufacturer is going to say that you can use the gun for something else, and its intended purpose is not to kill pandas, but then imagine you find that they've actually been shooting pandas with the gun to test it. Do you see why youtube-dl has put itself in a somewhat unfortunate spot?
I know my gut reaction to all things IP and copyright (and US) law causes me to lose all rationality, but I'd imagine there's a case to be made that test code is exempt through fair use. I don't think it could be made in this case for the reasons you highlight, but in general as long as the offending tests are a small part of the whole they should be considered separately to the actual code. Like if all my test cases are cracking some DRM then a takedown is probably fair enough, but if I'm using 3 out of 70 plus tests to make sure I handle some YouTube specific algorithm in my tool to download YouTube content then the actual videos I use don't seem relevant. But I get why legally its going to be a huge issue. I just wonder if there might be some case in future where test vs library code is a factor.
Edit: for example in the situation where one of their other tests just downloading a normal video used a copyrighted example instead of a CC example that shouldn't be any different, the content is irrelevant due to it being a test case, it could be any licence, it's just a test, the intent of a test should count.
