I used to think standardized tests were the way to go, but...
My kid recently went through another testing gauntlet, which is the 11+ test. For those who aren't in the Southeast corner of the UK, this is a multiple choice type test that looks a lot like an IQ test in some ways. It's a single sitting and determines whether you are allowed into something called a Grammar School, which is just a selective school, and the test is taken the year the kid turns 11.
So the thing about this test is that nobody will admit to it, but everyone in the middle class will hire tutors to help their kids pass the test. Furthermore, private primary schools will spend a lot of time prepping for this test in the two years before the kids take the test. Plenty of people do both: send their kids to a fee-paying primary school, and pay for tutors.
Guess what, the pass rate for fee-paying (aka independent) schools is way higher than for state schools. It's hard to break down which parents went for state + tutoring, but it's not exactly a stretch to suppose those kids did better than state school kids with no tutoring. Furthermore state schools that do well tend to be in certain wealthy areas.
If you have a look at one of these tests, it's pretty clear you benefit from practicing. Like just about any test, if you've done it before you are at an advantage, the more the better.
I don't see how this isn't gaming the test. The idea with a test is to uncover which kids will get the most out of the selective school, but how is that going to work when a select few are prepped for the test?
Now just for the record I did the SAT back when I was a kid. It's pretty much the same as the 11+, for a slightly older age group. Again, it doesn't make any sense to say it's hard to game. People are gaming it as we speak.
I think what might actually be hard to game is teacher recommendations. After all these people have been with the kid for a long time and know what they can do. They also tend to be distanced enough that they don't have to say good things about every kid.
I think what might actually be hard to game is teacher recommendations. After all these people have been with the kid for a long time and know what they can do. They also tend to be distanced enough that they don't have to say good things about every kid.
When I was 11, I had a teacher lean over during class and tell me how much she didn't like me, that I was the laziest student she had ever had, and that she couldn't wait until the end of the year because she wouldn't have to see me any more. To be fair: I wasn't a horrible student, was usually quiet without many friends, and just wasn't at the point to handle 2-3 hours of homework, probably due to undiagnosed issues.
I learned that teachers don't always have your best interests at heart, and I couldn't trust any of them much. I couldn't imagine the stress of trying to impress teachers to the point of getting recommendations, and I can't imagine she'd have given me one and I probably would have been scared to ask the others after that. On the other hand, it also became quite clear that you could game the system a little if you got the teachers to like you.
Just because they don't have to say good things about every kid doesn't mean that all of them will say things that give a fair assessment, though a lot would try a little more than this woman did.
Yeah this is something I thought about quite hard before I moved countries. I didn't want my kid to be at the mercy of some teacher who might not like him.
So then I ended up where we have this one time test, which stresses the hell out of everyone. Luckily he did well on it, school decisions came out last week and he got what he wanted.
I'm not actually sure which is better, part of why I thought maybe teacher recommendations would have been better was that my kid would undoubtedly get recommended if that were the system, he just happens to have that thing that teachers recognise. But even so he could have had a bad day and not made it in the system we actually live in.
How is hiring a tutor "gaming" it? They spend extra time learning, and as a result they do better. It's not like the tutors secretly give them a cheat sheet list of all the test answers; they still have to answer the questions the same as everyone else. If somebody puts more time into studying, whether on their own or with a tutor or with their parents, they'll do better than somebody who didn't.
Prepping for a test doesn't give you any practical knowledge or expand how you think about things; it's just about getting familiar with the test’s format and the way the question authors think. It'd be a bit like getting “better” at poker by always playing with the same people and learning their tells. It's “gaming” it because you're not better at what the test is supposed to measure, you're just better at what it actually does measure.
Prepping for the test is a valuable signal when you're talking about the entire pool of high school students. And it does not gate opportunities by economics as much as you'd think: https://www.nytimes.com/2017/10/25/magazine/asian-test-prep-.... Immigrant families eligible for reduced price lunch are able to scrounge up the money for these tests.
Give ten kids from the same class a self study online course and give another ten a private tutor, and they won't see the same score distribution. Where's the signal there?
SAT results don't have a line item that notes the amount of wealth or privilege that went into preparing.
> Immigrant families eligible for reduced price lunch are able to scrounge up the money for these tests.
Some families can't. Other families aren't aware, or aren't interested. But we judge the kids in the family for that.
That said... I don't know how the _new_ system will work at fighting that privilege -- there are still lots of ways for it to disguise itself. But we have to at least acknowledge the issues with the SAT.
But, to me at least, this goes beyond privilege. This is about diversity of skills and diversity of learner profiles and moving away from linear quantification of potential.
The effects of studying "for the test" as you put have been measured, improved test taking skills tends to be worth ~30 points which is not that significant. This matches my anecdotal experience and that of people I know who run SAT prep courses.
It's far more effective to actually teach students the material, either by teaching them new concepts or by firming up their understanding to ones they've already been exposed to. Particularly in Math, many students in high school have shaky understandings of fractions or algebra. Firming up these foundations can often lead to >100 point increase (given sufficient lead time). Those foundations are something the test is actually looking for since numeracy and strong algebra skills are a strong predictor of success in Calculus.
It's true that tutoring grants unfair advantages but this is going to be true in any system that uses skills as part of a selection criteria.
> The effects of studying "for the test" as you put have been measured, improved test taking skills tends to be worth ~30 points which is not that significant. This matches my anecdotal experience and that of people I know who run SAT prep courses.
I see this often but I suspect that it is lumping "Took a prep class for 1 hour on a Saturday" and "Spent 6 hours a week for 52 weeks with a tutor" in the same category.
Any tutor who only gets a 30 point increase won't be seeing much business among the folks I know.
However, I do agree with you that firming up skills is a remarkably quick way to get a significant boost. Being able to add 2 + 2 and come up with 4, repeatedly and accurately is often a big deal on these tests even with a calculator.
I'm familiar with that 30 point differential because it shows up in research.
You're right that 30 points isn't that much if you're thinking about the whole distribution, but I guarantee it can be significant around the selection threshold. That threshold might be implicit or explicit, but it's there, and if it's enough to nudge applicants past it, it's significant.
Sure, but having read a large chunk of educational literature I'm not aware of any alternatives with fewer distortions from parental aid. Grades correlate more highly with a good home life than test scores do, for example.
As long as we have "prestige" universities there's going to be some form of skills testing, and no one has ever designed an un-gameable test that can be administered nationally. The question we have to ask ourselves then is how we can reduce game-ability and I doubt we can make improvements that are more than incremental.
The JEE solved this problem by changing the format of the test each year. It's not disclosed before the exam. So it is really hard to form meaningful strategy that consistently helps you
> That said... I don't know how the _new_ system will work at fighting that privilege -- there are still lots of ways for it to disguise itself. But we have to at least acknowledge the issues with the SAT.
I'm in favor of using tests like that SAT as cheaper diagnostic tests, to help with student placements and accommodations, not for admissions. It's too bad this is being lost with the removal of the testing requirements, but I guess it doesn't matter much as the tests were never used this way in the first place, despite providing this information. https://cepa.stanford.edu/sites/default/files/ACT%20Paper%20... (note that I don't agree with the conclusions of this paper, merely the identified diagnostic criteria)
> This is about diversity of skills and diversity of learner profiles
I might believe that if I didn't believe that the diversity would mostly be token, with the majority of students in selective schools fitting a handful of templates.
> I think what might actually be hard to game is teacher recommendations. After all these people have been with the kid for a long time and know what they can do. They also tend to be distanced enough that they don't have to say good things about every kid.
How can you compare recommendations from different teachers? The whole point of the SAT is that it is a single number: you can compare SAT results from different schools and see which children are better at taking the test.
> If you have a look at one of these tests, it's pretty clear you benefit from practicing. Like just about any test, if you've done it before you are at an advantage, the more the better.
Yes, but you can't get from 500 to 1500 by practicing. You can probably get from 1450 to 1500, but that's just a few practice sessions to learn what the questions look like, how to fill in the answer sheet, how to tackle the most common ones.
If you’re trying to gauge what students will get the most from special schooling, the ones who prepped and studied for the test are probably the ones who are going to do the same if afforded the special school.
Teacher recommendations are incredibly easy to game. I asked my teachers for recommendations and, to help jog their memory and for their convenience, provided them a sample letter of recommendation that I wrote. They could sign and send that one, use the content as a reminder to write their own, or start from scratch.
I’m quite sure that money or services have changed hands for a recommendation in the past, probably more frequently than stand-ins have taken standardized tests.
Wouldn't teacher recommendations just leave to bribery? And not even that expensive. Have the kid deliver some nice present before each suitable occasion. Maybe brown envelope with cash.
At least with tests you could release previous tests and then possibly some groups could release free instructions on key points or tactics.
Teacher recommendations is how the U.K. used to work. It might be interesting to see what the arguments were at the time, though grammar schools feel like they’ve been a hot topic for quite a long time now.
I don’t think teacher recommendations are great. Presumably parents who currently pay for test prep will instead throw money at getting good teacher recommendations through the right schools, teaching good manners, etc. If I look at my mother and her sister, one went to the grammar school and the other didn’t because they had gotten a new headmistress who didn’t really know the students. The grammar school stopped being selective after a few years anyway (did I mention that grammar schools have been a hot topic for a long time…) and I think the teachers struggled with the change. Perhaps another difference is that grammar schools are often single-sex which I think has a bunch of benefits and drawbacks.
>I think what might actually be hard to game is teacher recommendations. After all these people have been with the kid for a long time and know what they can do. They also tend to be distanced enough that they don't have to say good things about every kid.
Recommendation letters aren't worth a lot. Most kids who have a shot at getting into a top-ranked school surely have some teachers they got along with. But then there's the luck of the draw whether said teacher will also put in an effort beyond a pro-forma recommendation.
I live in the southeast of the UK and my daughter took the 11+ and passed. She also went to a private prep school.
Private schools don't give as much 11+ help as you would think - if a child passes the 11+, the private lose all the potential senior school fees.
Tutoring for the exam definitely helps, but it's mostly exam technique and practise on those types of logical reasoning questions. They can't do the questions for you.
Well if you look at the scores, about 25% of a year group passes the test. It's something like 16% of state school kids and 36% of independent schools. Unfortunately the guy who compiled these stats closed down his website, so it's not so easy for me to look up.
The variation is enormous though. My kid's school is boasting about 95% of their kids passing the test. I know they've been prepping the test because well, that's what the kid has been doing for past couple of years.
> if a child passes the 11+, the private lose all the potential senior school fees.
It's true that some schools try to discourage the kids from taking it for that reason, but of course that's only true if there's a senior school section.
In the Netherlands, teacher advice (based on standardized tests)is the way. They really restrict practice for the test in multiple ways, some legal and some cultural.
Interestingly, the Netherlands tried it a few years without the results of the standardised tests being known to the teacher (the results came in after they'd given their advice). It turned out that children from a poor background started getting worse recommendations...
As a result that practice has been reversed and teachers now use the result from the standardised test again when making their recommendation.
My kid recently went through another testing gauntlet, which is the 11+ test. For those who aren't in the Southeast corner of the UK, this is a multiple choice type test that looks a lot like an IQ test in some ways. It's a single sitting and determines whether you are allowed into something called a Grammar School, which is just a selective school, and the test is taken the year the kid turns 11.
So the thing about this test is that nobody will admit to it, but everyone in the middle class will hire tutors to help their kids pass the test. Furthermore, private primary schools will spend a lot of time prepping for this test in the two years before the kids take the test. Plenty of people do both: send their kids to a fee-paying primary school, and pay for tutors.
Guess what, the pass rate for fee-paying (aka independent) schools is way higher than for state schools. It's hard to break down which parents went for state + tutoring, but it's not exactly a stretch to suppose those kids did better than state school kids with no tutoring. Furthermore state schools that do well tend to be in certain wealthy areas.
If you have a look at one of these tests, it's pretty clear you benefit from practicing. Like just about any test, if you've done it before you are at an advantage, the more the better.
I don't see how this isn't gaming the test. The idea with a test is to uncover which kids will get the most out of the selective school, but how is that going to work when a select few are prepped for the test?
Now just for the record I did the SAT back when I was a kid. It's pretty much the same as the 11+, for a slightly older age group. Again, it doesn't make any sense to say it's hard to game. People are gaming it as we speak.
I think what might actually be hard to game is teacher recommendations. After all these people have been with the kid for a long time and know what they can do. They also tend to be distanced enough that they don't have to say good things about every kid.