I think it's more avoiding clinging to temporary joy. Have you met a person who seems to live in their glory days, clinging to the last time they were happy? Spending their entire life chasing after those first highs? I find it hard to describe people like this as anything but 'suffering'.
I would also note that the meditative techniques of the Buddha eventually lead to an experience called the 'jhanas', which are pleasurable all on their own:
> Directed thought, singleness of preoccupation, and evaluation act as the causes. When the causes are fully ripe, results will appear — (d) rapture (piti), a compelling sense of fullness and refreshment for body and mind, going straight to the heart, independent of all else; (e) pleasure (sukha), physical ease arising from the body's being still and unperturbed (kaya-passaddhi); mental contentment arising from the mind's being at ease on its own, undistracted, unperturbed, serene, and exultant (citta-passaddhi).
There are plenty of joyful feelings in the path the Buddha set forth; spending time in meditation in these altered states of consciousness is part of the path, as I understand it. It's not all as austere as it may seem at first!
