I've found many of the claims of collateral damage to be from sites that I don't honestly feel were adding much value to our lives, even if they weren't strictly content farms.
My site that was penalized is The Online Slang Dictionary. http://onlineslangdictionary.com/ It's the oldest slang dictionary on the web and had the only freely accessible slang thesaurus on the web until about a month ago. In contrast to other slang dictionaries, The Online Slang Dictionary is only for legitimate slang words.
I've never used any black-hat or even slightly questionable SEO techniques or means to publicize the site. It started being penalized April 11, which is the launch date of what (IIRC) is being called "Panda 2.0". I've made many changes in hopes of fixing whatever it is that's causing Google to decide that the site is low quality, but haven't seen any improvement in ranking since then. A (now outdated) list of those changes is here: http://news.ycombinator.com/item?id=2664346 .
I guess my site was hit by Panda too.. I said guess because all of us are guessing since Google does not bother to inform us.
Like you, searching for a keyword, previously lands you in the first page - now it's on the 4th/5th page.
Try searching your site on Bing, Yahoo and see..
videowatchr.com uses Google video search and the youtube API to show youtube videos. So of course, it will have duplicate content.
If you use an API, you are getting someone's content most likely, does that make you a content farm?
How about if you use an RSS feed?
Google should publish guidelines on this Panda - so we're not guessing.
If @WalterGR above is correct, then changes to your site will only help when panda is rerun between one bot visit and the next (ie it's a batch process not a constantly evolved ranking).
If that's true then an awful lot of sites I have heard about may have been wasting time SEO'ing for nothing. (most obvious is serverfault)
Does anyone know if it really is an infrequent labelling run or an algoithm change and or it's run frequency?
I'd be interested to hear your specific story?