I've used it and it is good. Highly recommended. The videos you get aren't super great quality but they're useful and the price is right. The only complaint I have is that I'm a tad jealous I didn't start this site.
A smaller sample means a greater chance that it's not representative. If 10% of the population is colorblind (or left-handed, etc), you stand a greater-than-50% chance that a random group of five people won't identify a problem for this segment.
A larger number of available testers means more fresh eyeballs, hence a better chance of repeat queries yielding accurate first-impression data.
Feedback from people whose job it is to evaluate websites may not be representative of the average site visitor. They may be more familiar with common site layouts or have specific pet peeves.
Probably depends on what you're looking for. When you're just starting out a site, just a couple users will point out all sorts of user interface weaknesses that'll keep you busy fixing for a long while.
I think there's room for both. Turkers might simulate the casual browsers who happen upon your site. There could be valuable data buried in what they do in a short period of time.
Ha, that's great. I was thinking more along the lines of a piece of software that runs alongside the Turker's browsing session and keeps track of what they see and what they do with the mouse cursor, perhaps augmented by a video of the user's face taken by webcam, if available. But even simple feedback -- as you have built -- is extremely useful.
We've used it in the past, and will in the future - BUT this is not rigorous usability testing of the sort that you'll get in the lab. They're self-selected and they don't do well with loosely defined objectives (even though they tend to be North American native English speakers).
If you have access to a Mac and can recruit your own volunteers (college campus, etc), I'd recommend getting Silverback - http://silverbackapp.com/ . Not only do you get the screen capture, you also get to see frustration/delight on users' faces. Plus, it's cheaper if you plan on doing four or more tests.
Also, do they have a "more users, fewer minutes" option? I know I'd rather have 15 users for one minute each, or seven for two minutes each. After that, most users will have decided to stay or flee any ways. It would be good to get a sense of that early decision-making, especially across a statistically-reliable sample.
Cool. They could just give their testers instructions to use the cursor as a pointer to where they're looking. Or you could put that in your instructions to them. It's a decent enough proxy for eye movements any ways.
Have you seen footage from an eye tracking session? A user can't explain or point his mouse to where he is looking. The eyes move quickly and somewhat unconsciously.
Can anybody share a video session that they themselves commissioned? I'd like to see how the quality of feedback compares with the "official" demo of tripadvisor.
P.S. Somebody else mentioned eye tracking - I think commercial solutions for this are pretty expensive, but I've seen a few hobbyists roll their own using low cost webcams, though I'm not sure how they fair in terms of accuracy.
P.P.S. And somebody else mentioned just embedding a screen grabber directly onto your website - I'm sure it's achievable but with a browser extension, i.e. Mozilla. I'll have a play around and see what I can come up with.
It's possible from Javascript without extensions. I did a prototype of this, and while streaming the mouse x&y coords to a server was trivial, the hard part was making sure that what the user is seeing is the same layout I'm seeing myself.
For example if the user has a larger font size, then the layout might be different, so those coordinates would not mean what I think they mean. Faced with this I abandoned my own project on this, but I noticed that sites like http://robotreplay.com/ have appeared, not sure how they dealt with that issue.
I really like this idea, and I'll probably use this for my own site currently in development.
Have you considered selling an embeddable recorder that people can use right on their site? Why or why not? i.e. I pay you $xx for a google analytics style plugin that I can embed and have any user that wants to provide feedback...
Just trying to understand why you chose to implement as it currently stands.
There would probably be demand for a more expensive, higher-quality service. Someone else in this thread mentioned adding eye tracking. It also might be useful to add multiple trials from random individuals and create some sort of aggregate statistics. These are both things that you could charge a large premium for.
What is cool about this site from a business perspective is that it is a low-capital, low-cost way to do things with just one person, much like girlinyourshirt.com
It's a cool idea, but I will be reluctant to use it. Sole reason being, the method they select personnas to do usability testing isn't transparent and I'm not sure how much useful feedback will be from those personnas.
Can our usability firm get a white label version of your service?
Yes, you can resell it to your clients with your logo (not UserTesting.com) so you can use your brand and set your pricing. Email us at resellerusertesting.com and we'll get you started.
