Among other things, it turns out there's a lot of fluctuation of how much "value" a teacher adds from year to year. Teachers in a very high percentile one year end up in the bottom 40% in another year, for example.
"At the current time, VAM may show promise for lower-stakes,
diagnostic purposes. Examples include identifying teachers who might be low or high performing so that follow-ups can be done to verify the VAM findings. Inferences would need to be circumspect because of possible bias or sensitivity to the measure, but they could be a starting point for administrators (such as principals or superintendents) to target teachers for more thorough review."
There's this, too:
"The research base is currently insufficient to support the use of VAM for high-stakes decisions. We have identified numerous possible sources of error in teacher effects and any attempt to use VAM estimates for high-stakes decisions must be informed by an understanding of these potential error."
This is far more circumspect than "it works well!" Nevertheless it sounds to me like teachers' VA scores are effectively treated as grades, which is not consistent with the report you linked, nor the EPI report.
In practice, this means "standardized test scores." Does anyone actually think that's a good metric
If it isn't, we've got bigger problems - specifically the fact that we have no ability whatsoever to evaluate whether a school is doing a good job.
In that case, we need to fix our measurement process and waste as little money on unproven methods in the meantime.
Among other things, it turns out there's a lot of fluctuation of how much "value" a teacher adds from year to year.
If this is due to statistical noise, then teacher evaluation needs to be done on a multi-year basis. If this is due to internal variation in teacher quality, then maybe we need to either control the variation or accept that teacher quality varies widely and is mostly out of our control.
Note that if VAM doesn't work, then all the statistical studies claiming to show that teacher quality matters for student achievement are bunk. All those studies make some attempt at VAM, and show the variation due to teacher quality is large. Are you asserting that this is the case?
In practice, this means "standardized test scores." Does anyone actually think that's a good metric? I'll bet you most teachers don't think so.
And then I see stuff like this: http://voices.washingtonpost.com/answer-sheet/teachers/new-s.... This links to an EPI study that casts a substantial amount of doubt on VAM: http://epi.3cdn.net/724cd9a1eb91c40ff0_hwm6iij90.pdf.
Among other things, it turns out there's a lot of fluctuation of how much "value" a teacher adds from year to year. Teachers in a very high percentile one year end up in the bottom 40% in another year, for example.
Actually, downthread you provided a link (http://www.rand.org/pubs/monographs/2004/RAND_MG158.pdf), which says among other things:
"At the current time, VAM may show promise for lower-stakes, diagnostic purposes. Examples include identifying teachers who might be low or high performing so that follow-ups can be done to verify the VAM findings. Inferences would need to be circumspect because of possible bias or sensitivity to the measure, but they could be a starting point for administrators (such as principals or superintendents) to target teachers for more thorough review."
There's this, too:
"The research base is currently insufficient to support the use of VAM for high-stakes decisions. We have identified numerous possible sources of error in teacher effects and any attempt to use VAM estimates for high-stakes decisions must be informed by an understanding of these potential error."
This is far more circumspect than "it works well!" Nevertheless it sounds to me like teachers' VA scores are effectively treated as grades, which is not consistent with the report you linked, nor the EPI report.