But then, more likely than not, your testing how long it takes someone to learn your stack, and not how well they can solve problems once they're using your stack.
(or in other words, total time taken to do n tasks is mn + b, where m is the efficiency when using your stack, and b is the time taken to originally learn your stack. b is relatively large, so with n=1, that's a very different thing than with n=100.)
"But then, more likely than not, your testing how long it takes someone to learn your stack"
It tests how long it takes somebody to learn a small part of it, certainly.
Would that not be relevant for a job where you intend for them to learn your stack?
"not how well they can solve problems once they're using your stack."
A properly structured test could (and, clearly, both of our cases, should) accommodate both - to begin with, the ability to find their bearings in code with which they are not familiar and subsequently to solve problems within that context.
I usually do tests like this that last 45 minutes.
>or in other words, total time taken to do n tasks is mn + b, where m is the efficiency when using your stack, and b is the time taken to originally learn your stack. b is relatively large, so with n=1, that's a very different thing than with n=100.
For clarity, I work at google and I'd argue that for all but the most trivial problems, it would take even a good candidate more than 45 minutes to get their bearings. Most new hires do multiple days worth of codelabs before sitting in their desks.
In my experience the time it takes for programmers to get their bearings isn't necessarily about the triviality of the problem - it's usually about how decoupled/isolated the code you are working on is.
Sure, but my point is that it really doesn't matter when everything is unfamiliar. Imagine I put you into a situation where you're using piper, bazel, gflags, and protobuffs instead of git, make, argparse, and json, its going to take time to get your bearings no matter what. You'll have to figure out 2-3 new syntaxes.
Sure, you can limit scope, to things with 1-2 files where everything is handled for you and no data transfer between systems, building, or commiting required, but then we're getting into a class of problems where your limited to relatively simple, logical issues with very well defined APIs, a class of problems that data structures fit very, very well.
(or in other words, total time taken to do n tasks is mn + b, where m is the efficiency when using your stack, and b is the time taken to originally learn your stack. b is relatively large, so with n=1, that's a very different thing than with n=100.)