The same kind of process works for pure dev jobs. So, you’re a Rails shop? Take a Rails application you’ve actually built and deployed. Carve out some functional areas from the application: remove the search feature, or the customer order updater. Bundle up the app with all its assets, so that a single “vagrant up” gives a candidate an app that runs. Have them add back the feature you removed.
How would you compensate for candidates that are great developers but are new to Rails, or whatever specific language/framework your company uses?
1) If you think that the language/framework is intimately important to the hiring process (I don't but I'm not judging) then you don't compensate for that. It's an important data point.
2) If you don't think its important, you don't worry about matching "idiomatic" practices. You don't judge on obvious library mismatches, etc. If you don't feel confident doing that. Don't have the challenge interact with any language specific code. Let them do it in any language you feel confident of judging.
It sounds like the candidate gets to do this at their own pace, from home. If so, a good dev shouldn't have trouble learning enough Ruby/Rails to figure this out.
Are you asking how do you judge if someone performs under time pressure? If so, my first answer is don't. If that is a serious part of your real world work, you are doing it wrong.
The less flippant answer is, you give them a deadline but it is both A) flexible in the face of their "real life" (ie if they tell you they are going on vacation next week deal with it) and B) it deals with realistic time horizons (ie a week, not 4 hours).
Sorry my post was not complete enough - what I meant was that if you have two equally good candidates, but one happens to have experience in Rails, that person is going to perform the coding test more quickly than the other candidate i.e. get a more complete / correct solution in a given amount of time. So I hope in this situation there are no time limits?
I never want to have time limits in these tests. Generally I prefer to tell the candidate "This isn't timed and we don't care how long it takes you. It tends to take x hours (where x better be less than 4) and most candidates get back to us in a week or so. If you can't do that, its completely fine, let us know when you anticipate having it done. This is just so we can judge where you are in the pipeline."
Do you mean time pressure in the way that conveys that the company has poor project management? If so, you should probably find some way to explain that up front (set expectations) and non-Rails people might self-select out.
How would you compensate for candidates that are great developers but are new to Rails, or whatever specific language/framework your company uses?