Forum software like vBulletin and Invision often has this feature built in, and I've used it on a forum I help run. Unfortunately, after writing four or five custom questions, I soon found server logs showing spam bots blowing through the questions in seconds -- I suspect that since this is a common enough strategy, it's worth their time to pay someone $0.10 to pick the correct answer, then save the question and answer pair in a database somewhere for future use.
I don't remember what forum software it was, but they'd render a number or word using all periods (kind of like ASCII art) and ask you to enter it in a text field. I wonder how well that worked.
Here is a rough silly first pass using figlet. [1] It just displays a word. Next I need to accept a post of that word and do something. Maybe another version will use a game.