The primary test that I put any kind of leisure activity through is whether or not I feel better in some way during or afterwards. Most of television, for me, doesn't do that; it's a convenient way to fill some time, but leaves me feeling generally less energized than when I started.
A very good test. I find that even TV can fit this requirement if it's the right kind of TV: A 60-minute episode of The Wire, for example, leaves my mind in a vastly more engaged state than the same amount of time spent watching a few minutes of one show, changing channels, watching a few minutes of another, channel surfing, and finally concluding there's nothing on.
Good point. It's not just what you're doing in your downtime, but how you're doing it. Anything that you can make challenging, relaxing and significantly different (from work) can actually leave you more able to focus, and therefore go towards increasing your productivity.