No offense, "my video collection" is vague. What are you trying to watch? With which applications? Like, do file file_name_of_file_which_doesnt_work and post it here, or on their forums. For MPEG2 and VC-1 you need a license key for hardware acceleration. [1] They already give you a H264 license with the device but I think the default ffmpeg doesn't have support for OpenMAX by default. So you gotta compile ffmpeg with that. Instructions are on the forums.
[1] http://www.raspberrypi.com/license-keys/