What are teaching evaluations good for?

Over at Adventures in Ethics and Science, Dr. Freeride evaluates a recent study of how student academic performance is correlated with teaching evaluations.

The study is Carrell, S., & West, J. (2010). Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors Journal of Political Economy, 118 (3), 409-432

Like other studies of the effectiveness of teaching evaluations, it shows that high evaluations of teachers are correlated with grades (good grades match with good evaluations). But it also yields some surprises, such as that the students of more experienced instructors in the first of several calculus courses had lower performance in that first course but better performance in subsequent courses (no matter who taught the later courses). One possibility is that inexperienced teachers are more focused on getting students through the course at hand but less skilled at (or less focused on) teaching higher-level, critical skills.

Here is part of Dr. Freeride's analysis:

To my mind, this is a more complicated situation than students picking up inadequate study skills or teachers just teaching to the tests. Students are often surprised that learning a subject requires learning a sequence of increasingly more sophisticated models, or increasingly more sophisticated analytical techniques or methods of approximation, or what have you. Learning the next chunk of knowledge in the line is not just a matter of adding more on, but also of recognizing the problems with the chunk of knowledge you learned before. This is a surprise to many students...

One conclusion of this study, that student evaluations of faculty performance don't indicate that the students have learned all that we want them to, is no surprise at all. This is part of why institutions that care about teaching hardly ever rely on student evaluations of teaching as the only source of data to evaluate faculty teaching. (At my university, for example, there is regular peer reviewing of teaching, and these peer reviews are important in retention, tenure, and promotion decisions.)

But can even peer review yield insight into how to teach in a way that sticks? At first, I was skeptical that a study of calculus teaching can show much about evaluations and philosophy teaching. But in one respect, we do have this same issue of continuity. Many of my students think that what they are supposed to learn is the course content. Well, sometimes it is... But for the most part, I hope they're noticing the moves I make, and the moves that the various authors make, the writing skills and the argument analysis skills. Our courses aren't sequenced because learning in one class about what Theodor Adorno had to say about pseudo-individualization will not be helpful in another class on internalism about justification in contemporary epistemology. Nonetheless, a student can learn to do things in the first course that will help her excel in the second course. I'm just not sure that those things (and how well they are taught) are what wind up getting evaluated.