Whenever these cases come up, we tend to look at it with a status-quo bias - assuming that of course this data exists, and it's a question of whether it should be "free" (whatever your interpretation of that term). But in reality, things like YBB+ can impact the existence of the data in the first place - in this particular case, the evaluations existing in as comprehensive a form as they do depends on Yale's explicit sponsorship (having professors hand out the eval forms, etc.)
I'm not sure what the policy implication is - there's no easy rule here. But it's worthwhile to keep in mind that "information wants to be free, Yale can't decide how it's used" (an attitude that's a bit of a straw-man, but not too far from what some have argued here) ignores the reality that the next time the Yale faculty debate whether or not to collect and collate evaluations (or whether to have a numeric score component), they may well decide not to. (Which, of course, is a principle that applies to other cases of data collection - sometimes it's best not to collect it, just because you don't know how it'll be [mis-]used...)
You sort of have a point; the letter explicitly calls out that they're reconsidering providing this data at all.
But... given an adversary's policy of "we will provide this data if and only if it goes unused", there's no reason to give that policy any weight in deciding whether to use the data. If you decide to use it, the policy will kick in and you can't. If you decide not to use it, it's not relevant that it remains technically "available" for other people to also not use. That's not a reason to consider what Yale wants, it's a reason to slam them for being weasels.
If the university no longer provides the data this seems like a good opportunity for the application developers to manage that data themselves. They already have the students' course information so they could email them at the end of term and get their own evaluations -- removing Yale from the equation entirely.
Also worth pointing out that it's in Yale's interest to collect and provide the data. If they collect it, they can ensure a more representative sample size, even if they can't ensure how that data is ultimately presented. If students start using a third party service, not only does Yale still not have control over presentation, the data collected will probably make the faculty look worse by virtue of the fact that disgruntled students are more likely to respond than happy students.
> sometimes it's best not to collect it, just because you don't know how it'll be [mis-]used
An example where this has happened: Some (many?) U.S. public libraries have stopped storing historic checkout data for patrons, in order to avoid having to turn the data over to a govt. agency. I recall being able to get a list of every book I had ever checked out. Not any more.
That's why, if you can, you don't let hierarchical authorities generate the data in the first place. It would be much better if students generated the reviews on a trusted third party website with liberal copyrights. That would remove from the authorities the power to control the student-generated data in ways that are detrimental to the students. It would probably be helpful if Yale did stop generating this data so as to increase the incentive to generate the data outside the official channels.
The problem with ratemyprofessor (and virtually any 3rd-party "review" platform) is the potential for participation bias. Students only participate when they 1. Know about the platform and 2. Have strong enough opinions that they feel a need to share. This means that the ratings are biased in favor of the exceptionally good and exceptionally bad. (throw in a little human psychology, and the bias moves towards bad reviews appearing proportionally more often than they should).
The advantage of having the data come from Yale is that they can mandate that every faculty member must take class time to conduct the student evaluation. This ensures that [nearly] every student's experience in every class will be considered in the final results, which gives much more legitimacy to the results.
Widespread distribution of the data can also affect the process that the data are supposed to measure.
When I taught a college course several years ago, it was conventional wisdom that student evaluations were primarily driven by grades, thus encouraging grade inflation. A popular website called "rate my professors" included ratings for "easyness" and "hotness."
Female teachers were advised to anticipate receiving one or two rape threats in each batch of anonymous comments.
To me, it's not so much an issue of "information wants to be free" as it is "information wants to be put to productive use." In this case, the university had data that some enterprising students found they could use to make students' lives easier. They didn't just appropriate information; they added value to information. They organized it, and channeled it toward a useful purpose. They built a better Bluebook.
Yale can decide how the information is used. But if someone within the Yale community comes up with a Yale-beneficial use for the information, Yale should embrace that use.
I don't think this was a free-speech issue, so much as a missing-the-boat issue. Yale should be nurturing its hackers. Yale has an excellent CS program (I say this as a Yale alumnus), but in recent decades, it's not been much of a countercultural or hackerish place to be. Yale is top-notch at producing lawyers and investment bankers. It needs to get better at producing hackers. (By many recent accounts, it's starting to get considerably better. Maybe this episode will inspire a rethinking of sorts.)
To collect this data, Yale hands out surveys and students generate the content.
Yale needs the students to gather this data but the students do not need Yale. They could just as easily aggregate this data on an external website. It seems that they even have the ability to auth against Yale servers.
This content is generated by students and is supposed to be used to improve the University for the students. If Yale doesnt trust its own students with their own data it opens up the question of what the relationship between Yale and its students is and whether Yale is the kind of place for kids who understand how technology is shaping democracy, privacy, and our culture in general.
As much as I would love to sit here carrying on about how Information Wants to be Free, I do appreciate where she's coming from. YBB forced to students to actually read course evals and apply critical thinking to their class selection. YBB+ wrongly suggests that the quality of a class can be reduced to a scalar value.
We've complained about this bitterly going the other way. How many app developers are frustrated when they get a one-star review that says, "This app couldn't sync to my online account unless I was connected to the internet. Changing to five stars when that's fixed"? How many Amazon shoppers have seen one-star reviews that say, "The package was damaged during shipping. The return was really easy and I got a replacement in two days, but I wanted it for an event that happened before the replacement came."?
As an app developer who has gotten those reviews, the reality is they don't actually matter. Since everyone gets them, the real question is how many happy and 5 star reviews you get from the rest of your customers. You don't have 3 stars because of the guy that put 1 star, you have 3 stars because of the people who gave it 3, or all the people who didn't feel it was worth it to put down 5 stars.
The situation is the same here, at my school when viewing aggregate evaluation data, I find it to be VERY accurate to the actual experience I've had in those classes, and I think that's the issue here. Yale is protecting teachers from being called out for being bad teachers.
I feel like rating systems should be median based. Or bottom 25th percentile based. Those asshats that put 1 star "because it doesn't do x, otherwise 5" ruin 4 other 5 star ratings. When the bottom 25th percentile would still be 5/5.
I fully agree with you. For example when I look at Amazon reviews I never look at 5 star or 1 star review. Always 2,3,4 - those come from people that actually had issues - some smaller some larger. Then I can decide: ok, I might encounter these problems but it's something I can live with.
If the performance of the professor in a class cannot be reduced to a scalar value, then it seems probable that the performance of a student in a class also cannot be reduced to a scalar value, and professors should be required to write paragraph evaluations of students instead of just assigning them something from an A to an F.
In other words, if reduction of complex performance to a scalar value is invalid, then the entire grading system that defines academic success is bogus.
My alma mater (UC Santa Cruz) does exactly this, all course were pass/fail and if you passed you got a narrative evaluation, which ranged from terse sentences that obviously encoded traditional grades as text ("He did very good on the midterm, and excellent on the final, and excellent overall.") to some that were several paragraphs in length and discussed specific merits of the work done in the course. Amusingly they also had a committee that would translate your narrative evaluations into numeric grades so you could apply to grad schools that required a GPA.
This is indeed exactly what happend at my lower/middle/high school--we received a full paragraph from each teacher evaluating our performance every trimester.
Doing so at a college scale would be too costly. But you do have all the grades that go into your average, comments on each essay, office hour time, etcetera. The nuances of your performance don't matter in your GPA, that is, your Grade Point Average, but you do have a bit more information available to you, yourself.
Actually, my experience in college was that classes could usually be reduced to a scalar value.
Regardless of subject matter, I learned the most in classes that had good teachers who received good evals.
However, the administration doesn't think that way; they think that it's more valuable to take the "right" course than to take a course with the right teacher.
There is already some data that Apple doesn't provide in the app store but that third party sites do collect, namely price history. Being able to see if an app often goes on sale is very useful to the user, but not information that Apple or the developers want to make obvious.
> In doing so, the developers violated Yale’s appropriate use policy by taking and modifying data without permission, but, more importantly, they encouraged students to select courses on the basis of incomplete information. To claim that Yale’s effort to ensure that students received complete information somehow violated freedom of expression turns that principle on its head.
I think she doesn't really know what free expression means. It does not mean, "you can say what you want as long as we agree with it." That's certainly how I read "more importantly, they encouraged students to select courses on the basis of incomplete information". Free expression means that the administration cannot control the message; people can say what they want. This is OK; win in the marketplace of ideas ("maybe I should evaluate my courses with more contextual information"), not by treating your students as children who can't decide for themselves how to consume information.
Also, this bit about reconsidering whether to provide the data at all is childish. "You're not interpreting our data the way we want so now you can't have any! I'm taking my ball and going home!" It's worth remembering (both here and in the thus-far imprecise and confused discussion of who "owns" the data) that the students are the ones providing the evaluations being aggregated here; at the very least, it's unclear why they'd be expected to continue doing so without getting some quid pro quo benefit from it.
Stop trying to control everything and trust your students to be smart and thoughtful.
Not everyone acts as rational machines, professors' egos have to be stroked too. They'd like to think that they can't just be reduced to a number between 1 and 5.
Besides which, I think exposing rating systems for courses creates perverse incentives. It's been well studied that students who get better grades rate courses higher. Is that really what we want to be measuring professors against?
OK, but now you're arguing that Yale students shouldn't be exposed to free expression because they're incapable of handling its consequences. I think that's silly, but fine: it's a real argument. But let's stop pretending this kind of paternalism is consistent with even a modest notion of free expression, because it's very clearly not. It says "you can only say things we approve of, because we know best." Being right about knowing best doesn't make it not a limitation on free expression. Free expression as a principle necessarily doesn't hinge on whether you think the individual expressions are right. That's, like, the whole point of it. [1]
You're also arguing that 1-5 rating systems are silly and Yale shouldn't bother collecting them, but that has nothing to do with the YBB+ website or how students will interpret it. For example, wouldn't those perverse incentives you mention be in place already, given that providing feedback to professors was the whole point? Ratings don't magically transform from useful evaluation tools to misguided oversimplifications just because students have convenient access to them. Either they carry useful information or they don't.
[1] I'm not taking a free expression absolutist position here. (I'm tempted to, but it's totally unnecessary for this discussion.) The point is that to the degree to which you make rules about what kind of thing people can express is, well, trivially the degree to which you limit free expression. And limiting the ability of students to transmit truthful data about the courses they're taking is a very strong limitation, and it would be hard to formulate a consistent, principled policy that permits this kind of restriction and doesn't just say, "you can't say things we don't like".
Why is this a free expression issue? Yale is simply saying you can use the data we collected, but in order to do so you must show ALL of it, including comments. If Yelp changed their open API rules and said you can display the star ratings for a restaurant, but you have to display the text of the reviews as well, would everyone be up in arms about it?
They're not talking about "free expression" in the generic sense. They're specifically talking about Yale's Freedom of Expression Policy [1]. Comparing this to Yelp doesn't make sense, because the Dean of Yelp never said, "I disagree that Yale violated its policies on free expression in this situation." The policy in question specifically states that Yale values "the need to be able to 'think the unthinkable, discuss the unmentionable, and challenge the unchallengeable.'" and acknowledges that "such freedom may sometimes make life uncomfortable in a small society such as a college. But it also asserts that 'because no other institution combines the discovery and dissemination of basic knowledge with teaching, few need assign such high priority to it.'"
I don't really have a bone in this fight, so to speak, but I do find it interesting that the Dean of Yale has such a problem with the way their students are using their data to express an idea, when their very own Freedom of Expression Policy says:
Yale's commitment to freedom of expression means that when you agree to matriculate, you join a community where "the provocative, the disturbing, and the unorthodox" must be tolerated. When you encounter people who think differently than you do, you will be expected to honor their free expression, even when what they have to say seems wrong or offensive to you.
Although, if we're being pedantic, their policy technically states that they expect their students and graduates to honor others' free expression, not that they agree to honor their students' free expression.
Yale students write Yale course reviews in order to see their grade. You get an email saying "a new grade has been posted," then you fill out the review form, then you see your grade.
That would be the definition of "freedom of speech" in many western countries. (or at-least i believe it is similar in similar countries other than my own, i have not fact checked that.)
For example in Sweden we have laws that provide "freedom of speech".
It means that one can ALMOST say what one wants to anyone. One may however not say/or behave in a way that is intended to offend a group of people.
Saying such a thing leads either to Jail or a Fine.
Freedom of expression and freedom of speech are synonymous, but I think the former is a clearer phrase. Freedom of expression is generally never absolute (e.g. the US has libel and incitement laws ruled consistent with the Constitution), but for what it's worth, I believe European laws against offensive speech are absurd and make a mockery of free expression. It's not really free if people liking it is a criteria for being able to say it.
The law, of course, is not in play here, because Yale is a private institution not wielding government authority. But the degree to which it supports free expression is exactly the degree to which it lets people say whatever they want.
Are you making a mockery about my country's laws?
I am deeply offended.
You know that, it is a criminal offense punishable by jail time to do such a thing.
I agree with the second paragraph, but not the third. It's not really childish to remove data that is being used incorrectly. Once the information is removed, then it becomes up to the people using it wrongly to gather that data, and then use it wrongly. Till then, you could be reasonably said to be trying to reduce harm.
Perhaps I'm missing something, but I do not see an apology behind that link. A detail of Yale's reasoning, but not an apology.
> However, I disagree that Yale violated its policies on free expression in this situation.
I'm not sure what "[Yale's] policies on free expression" are, but my understanding here is that Yale blocked the IP address of a site because it was using some data that Yale published, and modifying it in a way Yale disagreed with. There might be an argument to be had under copyright, but I don't think it's important at the moment. Yale states this:
> the developers violated Yale’s appropriate use policy by taking and modifying data without permission, but, more importantly, they encouraged students to select courses on the basis of incomplete information.
Abusing the fact that you have power over the students' access of information to prevent access to information you disagree with is what people have a problem with; this is especially true when you label something like this "malicious activity". These concerns are not addressed by the dean's letter.
I wonder if they act the same way about contentious academic papers. Prof A writes a paper, Prof B writes a rebuttal quoting Prof A's paper, Prof A cries copyright and shuts down Prof B?
I think this is a great example of both compromise and overreaction. In the end, they are clearly trying to give students the tools they want, but this should have been the first response, not the second. I can't find it now but the other yale student that wrote the chrome extension was the main catalyst. He took the complaints of Yale and removed them therefore forcing them to find new logic, which they did. Hats off to him.
I did get a chuckle at this backhanded positioning in the very first paragraph (which I highlight in _ _ below). Kind of set the tone of what I expected in the note, but was disappointed to see:
"In retrospect, I agree that we could have been more patient in asking the developers to take down _information they had appropriated without permission_, before taking the actions that we did."
The entire note has an odd, arrogant read to it. If the new tool in fact provided complete comments as well as numeric scores, Yale would have cheered and saluted them ... but because they didn't, Yale shuts it down? Is that really the issue?
If by "apologizes" you mean "does not apologize", I guess so. ;-)
Universities tend to move very slowly, and I get that these situations can be hard for them, but the whole letter is disingenuous, at best. AFAIK, the students aren't guilty of "modifying data without permission", nor did they "encourage[d] students to select courses on the basis of incomplete information." That's like saying no one should use a course catalog because it doesn't have complete information. Of course scalar averages don't capture the entirety of the feedback on courses. But I bet there's a meaningful difference between 4 stars and 2 stars, or whatever the rating system is, and I'd like to think that Yale students are clever enough o understand this, and to be able to go through more detailed evaluations when it makes sense to them.
If the Dean had just said "holy crap, we didn't realize how easy it is to do this and we're freaked out about it, we need you to close this down for now, but please help us build the next generation system", people would have a very different reaction.
To that end, the Teaching, Learning, and Advising Committee, which originally brought teaching evaluations online, will take up the question of how to respond to these developments, and the appropriate members of the IT staff, along with the University Registrar, will review our responses to violations of University policy.
Did no one else feel that the biggest issue with the original ip block was not the motivation for the block but rather Yale's blocking of a website on copyright grounds in the first place?
Maybe Yale is indeed in the right to try to control use of this data. But Yale most certainly is not in the right to unilaterally block access, regardless of the reasoning.
(NB., my concern is not with legal rights, but rather a higher standard of moral or ethical rights that Yale, as a purportedly high-minded institution, should hold itself to.)
Welcome to the 21st century. That single minor incident now escalated into a news item of several days. Lot of lost PR points. All because they feel the need to control the course selection process. Sad.
> All because they feel the need to control the course selection process.
It's almost as if the entire purpose of their institution was controlling the course selection process. You know. By offering courses in the first place.
Indeed, I think that's what Miller is referring to when she writes in the 2nd-to-last paragraph:
Just this weekend, we learned of a tool that replicates YBB+'s efforts without violating Yale’s appropriate use policy, and that leapfrogs over the hardest questions before us.
The most interesting thing to me about these policies is that the Freedom of Expression Policy explicitly states how crucial it is to be tolerant and accepting of people's ability to express ideas that are "provocative, the disturbing, and the unorthodox," and that this means they value the ability to "think the unthinkable, discuss the unmentionable, and challenge the unchallengeable." And yet, they are having a rough time because some students used their data to express an idea in a way they hadn't intended or thought was possible.
At the same time, I actually couldn't find a single clause in their Appropriate Use Policy that had been violated. The only restrictions I saw in the policy were that their data be used only for non-commercial and academic purposes (which it was). I didn't see anything about only being able to use data in its intended manner. But maybe I'm missing something (for example, maybe the actual YBB site has its own Appropriate Use Policy separate from their IT policy).
EDIT: I also just realized I was looking at the Quick Reference for their Appropriate Use Policy, so maybe there's something else in the full text that was violated:
I can't find the apology in that letter. I see stifling:
* "we need to review our policies and practices" -- We're going to make it policy that these ratings are gone.
* "We will also state more clearly the requirement/expectation for student software developers" -- The requirements will become more burdensome and bureaucratic, and we will be able to better cover our asses.
* "and we will create an easy means for them to do so" -- (I don't buy this one.)
Yea I felt like this was sort of a: Hey we acted too quickly, but we understand you like this. It's still not what we want you doing, and you found away around our policies. So we are probably just gonna take it all down and not give you any information.
It's almost certain that Yale will no longer make any course/instructor review information available online as a result of all this. The Dean as much as telegraphed this in his missive.
Censorship is not a valid tool for an educational institution to use. They are free to promote their version of the information, ask people to not use the other, etc. but however much they dislike it, blocking it should not even be considered.
Which brings up an interesting question. Does Yale exist because it's an educational institution, or does it exist because it's a money-making enterprise?
Can it be both?
Coming from a country where higher-education is covered by the government, I think it's a conflict of interest for an educational institute to be in the money-making game
If the dean is suggesting that there are limits on the value of single-measure evaluation, I look forward to hearing about what she has in mind for student evaluations. I hope and expect that she will devote the same amount of energy to our long-outdated grading system as she devotes to hiding information about professor evaluations.
It probably has to do with how the YBB+ portrayed some teacher in a bad light. The integrity of those 'excellent' professors and tenured careers has to be protected.
I wish my university even had all those reviews we do at the end of the year available for all to see. They don't do sh*t with it. I once wasted my time by suggesting my university create an API for students to easily get any data about the school. B/c then we would all see how 'great' some of our professors are and what we each think about them.
Sure 3rd party website like 'ratemyprofessor' are good, but they don't have the ease of acquiring rating data like schools do that pass out reviews to each student at the end of a semester. They have to protect the 'wizard of oz' lest we all see who really is behind the curtain.
entrenched bureaucratic culture and a wide gap between the students and the faculty.
a post facto weak-sauce apology is about what I expected here. I get the distinct impression that what the Dean regrets most is that the story was widely publicized, rather than that her initial reaction was "heavy handed".
Well, they should have contacted the developer of the interface beforehand. If that was not plausible, they should have given all the students a heads-up a suitable time before blocking.
If they had done all the above, then I don't know what the fuss is all about.
This is a pretty big setback for Yale in terms of refocusing more on STEM. But they're in good company. I'm sure 10-15 years ago music company executives were similarly pissed off.
I think that, in considering their options, "Yale's stance on a fly-by-night improvement to the course catalog" has a very low weight, and will be forgotten by next admissions season.
Because Yale has options too - if a few students decide to make a principled stand, there are others waiting in line just as good who care slightly less.
I'm gonna bet that the advisory committee is going to be made up only or at least largely faculty and administration. If they want to do things right, any such advisory committee on the matter should be approximately 50/50 students and faculty with administrative people there to listen and figure out how to overcome and legal issues that may exist that would inhibit the solution that faculty and students come up with together.
Locking down data and requiring that students get approval before working with it fundamentally violates the freedom of intellectual inquiry that any healthy university requires to function.
Indeed. There seems to be a big disconnect here from Yale. They collected this data and made it available. However the Dean and Yale's main problem seems to be [1] that YBB+ wasn't showing all of the data. That the numerical ratings are some how strongly linked by spooky action to the detailed reviews and that you cannot show one without the other.
This is, of course, crazy.
From simply a UI/design position, you can't easily display/sort on X different professors/courses each of which has 1..N detailed reviews that are blobs of text.
From a broader perspective sometimes you need to decouple data to be able to better process it. In fact, given so many students used YBB+ it is reasonable to conclude that YBB+'s presentation of the data is doing a job informing students than Yale's own systems.
Everything is a remix
[1] From the article: "The tool created by YBB+ set aside the richer body of information available on the Yale website, including student comments, and focused on simple numerical ratings. In doing so, the developers violated Yale’s appropriate use policy by taking and modifying data without permission, but, more importantly, they encouraged students to select courses on the basis of incomplete information."
The problem is whether it was doing a good job. Sure, students may have felt they were making good decisions by looking at 1-5 average ranking, and yes, the UI was very pretty and made it look like it was providing you lots of information, but the fact is, a simple average of a single metric is not a good way to pick a course.
There are always hard, scary courses, and those classes are going to end up ranked lower because the material is harder or the professor isn't as "cool" or because a 20-year-old doesn't see the point in learning how to do proofs. Letting that become the dominant factor in course selection is not good for education (or intellectual freedom--what happens when a professor with unpopular views is ranked poorly and his/her classes become sparsely attended?)
well you could do NLP on the reviews and sort it using some lexical grammar. if you're into sorting and all that.
mort importantly the ybb reviews are basically useless from a reality perspective due to the way the data is collected (gateway for letter grades) -- they never told you anything important no matter their format, and never will.
if you want to know what a professor is like you just ask around or drop in on the class during shopping period. ybb+ is not an improvement on that process.
digital equivalent of 'dropping in': coursera or open yale. digital equivalent of 'asking around' ... ? not ybb+
personally i think it would make sense to get each professor to record a video introducing themselves and the material. then skip that awkward first lecture. actual teaching video would be tops...
As I read the letter, first they're going to stop providing faculty evaluations, and then they're going to set up a bureaucracy overseeing student-developer use of university resources. After that, they may or may not reallow the app, but as the letter points out:
> Just this weekend, we learned of a tool that replicates YBB+'s efforts without violating Yale’s appropriate use policy, and that leapfrogs over the hardest questions before us.
Setting up bureaucracies = we're going to take a long time to figure this out, hopefully long enough for everyone to forget this. Then our crappy committee will get nothing accomplished, except this smokescreen act of course.
I'm not sure what the policy implication is - there's no easy rule here. But it's worthwhile to keep in mind that "information wants to be free, Yale can't decide how it's used" (an attitude that's a bit of a straw-man, but not too far from what some have argued here) ignores the reality that the next time the Yale faculty debate whether or not to collect and collate evaluations (or whether to have a numeric score component), they may well decide not to. (Which, of course, is a principle that applies to other cases of data collection - sometimes it's best not to collect it, just because you don't know how it'll be [mis-]used...)