This is a problem with the recent social movements -- and you can even listen to Dave Chappelle offer this criticism: the people who may be right may be entitled to be offended, but the way they're going about it is not going to help the cause.
It is not good, when anyone who offers even a little bit of truth or reasonableness during a discussion is struck down as the enemy and attacked instantly by the vehemence of someone's offense.
If the successful chief scientist of a huge company can be taken down by forces such as this, while speaking truthful statements, what is the likelihood if I were a lowly junior researcher, I would dare voice an opinion or participate in the discussion? Or after a while, seeing what happens to such people, want to help?
Fear and exclusionary self-righteousness are not the way to win a social struggle. History shows that success comes from making more and more people feel they are a part of the solution -- not making them fear to open their mouths.
Especially if, in the end, we each still stand alone in the voting booth and pull the lever based on how we've been made to feel.
Well, cancel culture is like a superpower for these social science professors and "thought" leaders, who otherwise might not have much klout or academic prestige. Of course they will use it liberally.
It's a kind of anti intellectualism, and ironically largely university funded.
It’s one of the few ways to deal with people who don’t act or talk in good faith. Certainly there’s many cases of it being overused, but in a way it’s sort of denying that there are actual direct people and forces that are fighting tooth and nail against progress.
From my perspective, discrediting the speaker to silence them IS an argument in bad faith. Whether its an ad hominem, tu quoque, or a pitchforking kangaroo court.
If someone is making a good faith rational argument, and the response is "I find your conclusion potentially offensive, so I will suppress your expression"; that is the suppressor acting in bad faith.
The energy around "doing what feels right despite evidence" and "the ends justify the means" have turned into the destruction of nuance. It's very hard to say "I agree with your conclusion, but not your reasoning" because how one arrived at a conclusion no longer matters, only that the conclusion fits the dogma.
I agree with your reasoning, but not your conclusion. I think there is a level to which the reasoning stops applying in the presence of trolls, which are absolutely bad actors. The conclusions a troll presents are not even necessarily ones that they even believe, and toxic words and actions also stifles intellectual debate. Flamebait SHOULD be suppressed, and thats fairly universally agreed upon. That a downvoted comment turns more opaque on this platform is evidence enough of that. Often times there's no intellectual debate happening online, just social posturing on opposed sides.
Once you start talking about real people - like say for instance a video is posted of some white woman harassing and calling the police on a black person. Those doing the "cancelling" by calling their office are actively trying to create a societal social boundary, where you can't do that without consequences because you'd lose the trust of people you work with. That people are shocked someone could be fired for such a thing means they don't agree with that social boundary.
Re: "doing what feels right despite evidence" and "the ends justify the means" - those are not at all what I'm espousing and I certainly don't agree with them.
Directly suppressing flame is different than discrediting a speaker to silence them.
I was mostly replying to the "it's overused" claim, that suppressing a bad faith actors may justify collaterally suppressing a good faith. I'm not sure that's what they were saying, but they didn't seem bothered by it.
There's nothing that constrains its application. Its users are those energized by outrage and offense and act on emotional reflex. Their position of pure victimhood is the perception which justifies their right to take down anyone for any reason.
It's single-minded, irrational, damaging to society and acts with little regard for the nuance of actual reality - which means it cannot in the long-term solve actual problems we face without creating bigger ones.
Acting is good faith and with intellectual honesty is always going to require voluntary participation. But if one contestant decides to knock the pieces off the table, then what we're watching is not a chess game, but one party claiming chess superiority because he bit his opponent on the neck.
Seems like a stretch considering there hasn't been a demonstrated benefit to this sort of antagonism, it all just exists to justify itself rather than to facilitate a dialogue. Much more likely is that the humanities are just becoming increasingly desperate to find relevance as they fail over and over to contribute anything new to society. It's far easier to invent gender frameworks and equity rhetoric than to actually solve problems like predominantly one parent households or the seething xenophobia and sexism of the trans community (Latin X imperialism, treatment of black comedians, bigotry towards safe spaces for women, etc)
Are you suggesting cancelers are commiting crimes?
Ultimately what you're trying to say, it appears, is that people who are choosing to speak out about injustice are bad actors because other people choose to listen to them.
I think you'll find that there are essentially no jurisdictions where voicing your displeasure about someone's actions is a crime.
In most US jurisdictions, Harassment requires the (credible) threat of violence. Otherwise it's just speech, and protected like any other speech. My understanding is that this is true in most international jurisdictions as well.
Do you consider yourself a supporter of "free speech"?
This concept is so fraught: if I say that I believe someone did a bad thing, that's acceptable. But if a bunch of other people agree with me, suddenly I've committed a crime.
"Cyberstalking is the act of using the Internet to systematically and repeatedly harass, threaten or intimidate someone. This can be done through email, social media, chat rooms, instant messaging or other online mediums."
"Cyberstalking is a federal offense and many states have cyberstalking laws. Cyberstalking falls under anti-stalking, slander and harassment laws that are already in place and are punished similarly."
"There have been a number of attempts by experts and legislators to define cyberstalking. It is generally understood to be the use of the Internet or other electronic means to stalk or harass an individual, a group, or an organization. [...] Stalking is a continuous process, consisting of a series of actions, each of which may be entirely legal in itself."
"Cyberstalking is a technologically-based "attack" on one person who has been targeted specifically for that attack for reasons of anger, revenge or control. Cyberstalking can take many forms, including:
1 - harassment, embarrassment and humiliation of the victim
2 - emptying bank accounts or other economic control such as ruining the victim's credit score
3 - harassing family, friends and employers to isolate the victim
So are are suggesting that "I don't approve of your opinion on this topic" is equivalent to "threatening her with rape and strangulation, publishing her home address and Social Security number, and posting doctored photographs of her.", to quote the wikipedia article you linked?
Or perhaps more specifically, can you present an example of cancelling that was done "specifically [...] for reasons of anger, revenge or control", and not due to other reasons, and that includes embarrassment and humiliation, economic control like ruining the credit score, harassing family and friends or employers, or scare tactics to instill fear?
I can't.
You trying to frame people being upset with someone's actions and expression views you disagree with as stalking is an interesting take, I'll give you that.
> Or perhaps more specifically, can you present an example of cancelling that was done "specifically [...] for reasons of anger, revenge or control", and not due to other reasons, and that includes embarassment and humiliation, economic control like ruining the credit score, harassing family and friends or employers, or scare tactics to instill fear?
Uh? Seriously? Just about any case of cancel culture.
You aren't reading carefully. I said and not due to other reasons. There are legitimate reasons to complain to someone's employer about their behavior.
Damore's employer for example wasn't really targeted. I know, I worked there during the event. I commented on his doc before it was leaked to the press. Coworkers complained to Google leadership because Damore's document made them feel unwelcome. Ultimately it was for that reason that Damore was fired.
Anger? No. Revenge? No. Harassing employers? Not really. Scare tactics? I can't recall anyone threatening Damore inside Google. It might have happened externally, but wasn't widespread.
The issue that I keep trying to point out is that you're taking people's legitimate airing of concerns and claiming that it is done illegitimately. Ultimately you can't see other people's intentions. Framing all grievances you disagree with as illegitimate and made with only ill intent is a questionable practice.
You can make the argument that everyone lied: that no one really felt unwelcome due to Damore's document, and that Google leadership agrees with you that he did nothing wrong but felt pressured to fire him anyway and that... But at that point you're taking your own preconceptions and applying them to everyone else, and assuming that only your view of the world is valid and that everyone who doesn't take the same meaning away from something that you do is acting in bad faith. That's not solid ground for claiming that someone is a criminal (you can apply the same reasoning and get "everyone who disagrees with me is a criminal", which I hope you'd agree isn't just).
> You aren't reading carefully. I said and not due to other reasons.
I read it and ignored it as a completely arbitrary constraint that you added.
> There are legitimate reasons to complain to someone's employer about their behavior.
Sure.
> Coworkers complained to Google leadership because Damore's document made them feel unwelcome.
This is not a legitimate reason. First of all, because no one can tell whether it's actually true, since it's an entirely subjective feeling, justified by an aberrant interpretation of what he wrote. Andf then because "makes me feel unwelcome" is ridiculous when it's the opinion of a single employee in a company of 120 thousand.
The objective fact is that a number of employees were angry at someone who expressed a political opinion they didn't like and mobbed their employer to have him fired. They can claim whatever subjective feeling as a reason for this.
> Anger? No. Revenge? No. Harassing employers? Not really. Scare tactics? I can't recall anyone threatening Damore inside Google.
So, you deny that the people who asked for him to be fired were angry? That's pretty curious. You believe they were "feeling unwelcome", a pretty vague feeling, but you aren't able to detect a primal, generic anger in their actions.
The firing can easily be interpreted as a revenge against someone who expressed a political opinion they didn't like. They mobbed the employers until they had to do something as drastic as firing him. And finally, they didn't threat Damore, they actually damaged him, implicitly threatening everyone else who might want to voice similar opinions.
> You can make the argument that everyone lied: that no one really felt unwelcome due to Damore's document, and that Google leadership agrees with you that he did nothing wrong but felt pressured to fire him anyway and that...
Exactly. And sorry, but I think that what happened is extremely plain, and it requires a lot of ideological contorsions to deny the anger, the harassment and the revenge, all based on vague purported "feelings", "asked for drastic action but not harassed", etc.
> is acting in bad faith
An individual acting in bad faith lies to others; an adherent to an ideology might lie to him/herself first.
> I read it and ignored it as a completely arbitrary constraint that you added.
Legally, it's quite an important constraint. If you're accusing people of criminal acts, you should, you know, actually be certain that the acts are criminal.
> This is not a legitimate reason.
Bluntly: of course it is. If I feel unwelcome in my workplace, I am encouraged to report it to my management. Psychological safety is incredibly valuable and is something that a lot of companies don't do enough to foster.
Put another way, if someone started posting swastikas around the office, I would be well within my rights to complain to the higher ups that those things made me feel unwelcome. There's no difference except that you feel one complaint is more legitimate than the other. They're still both complaints, made in good faith by those making them.
> So, you deny that the people who asked for him to be fired were angry?
Sure they were angry. But anger was not the specific reason the complaints were made.
> The firing can easily be interpreted as a revenge against someone who expressed a political opinion they didn't like.
Indeed, one is free to interpret the events that way. But that interpretation is no more valid than any other. In fact, it might be less valid given that courts ruled that that isn't why he was fired ;)
> They mobbed the employers until they had to do something as drastic as firing him.
Google was free to not fire him. My understanding is that it was debated, and that eventually executives decided that ethically firing him was the right decision. Could they be lying? Sure. But what makes you so certain that executives agree with you, but feel that their hands were forced? Is it so really unbelievable that a company might feel that it was ethically correct to fire Damore?
I'm not asking if it's harder for you, personally, to face that. Clearly you don't want to believe that, but your unwillingness doesn't make it false. Like, clearly a lot of people thought Damore did a bad thing. So many in fact that there were, to quote you, "mobs". Isn't it feasible that some of the decision makers were among them?
> and it requires a lot of ideological contorsions to deny the anger, the harassment and the revenge
But once again, the question isn't what you believe their goals to have been, it's what they believe their goals to have been. If the goal wasn't to harass, wasn't to get revenge, but instead to improve their workplace, it's not criminal. And I'll reiterate: courts have, on more than one occasion, ruled that Google was within their rights to terminate Damore for the stated reason of making others unwelcome.
You're creating a sort of thoughtcrime: if I complain about someone for the wrong reasons, I'm actually harassing the person that I'm complaining about, and I should go to jail. Also I don't get to decide what the wrong reasons are. You do. That's not just.
> If you're accusing people of criminal acts, you should, you know, actually be certain that the acts are criminal.
Intention matters, and that is usually left to the interpretation of the events. A judge or jury is free to decide what is the probable intention behind an action, whatever is the reason given by the accused.
> If I feel unwelcome in my workplace, I am encouraged to report it to my management. Psychological safety is incredibly valuable and is something that a lot of companies don't do enough to foster.
It's true, but to claim that the politely expressed and abstract opinion of someone makes you feel unwelcome is a stretch. And then, what about the feeling of being welcome and psychological safety of all those who might have a similar opinion? Which opinions this applies to? Am I free to go to the management and say that all those who speak of "white privilege" make me feel unwelcome and I want them to be fired?
> if someone started posting swastikas around the office, I would be well within my rights to complain to the higher ups that those things made me feel unwelcome
I don't really get this "makes me feel unwelcome". Being welcome or not is something between you and the company, not a single one of your coworkers. What you mean is that someone is a jerk and is repeatedly posting highly objectionable content. He might be then admonished to stop. End of the issue.
> Google was free to not fire him. My understanding is that it was debated, and that eventually executives decided that ethically firing him was the right decision.
Of course one can ignore the political pressure, media articles, and very vocal minority who asked for him to be fired. But then it becomes a different story.
> the question isn't what you believe their goals to have been, it's what they believe their goals to have been. If the goal wasn't to harass, wasn't to get revenge, but instead to improve their workplace, it's not criminal
If my idea to improve the workplace is to have all those who express feminist ideas to be silenced or fired, because as a male they make me "feel unwelcome", what do you think? Is it all fine?
> if I complain about someone for the wrong reasons, I'm actually harassing the person that I'm complaining about ... Also I don't get to decide what the wrong reasons are.
If I complain about the communists or the Jehova's witnesses because I don't like them, and I try to pressure the management into firing them, I am the one to be fired, not them. And I don't get to decide what the right reasons are. Seems simple to me.
> What you mean is that someone is a jerk and is repeatedly posting highly objectionable content.
The irony here being that this describes the Damore situation precisely.
> Am I free to go to the management and say that all those who speak of "white privilege" make me feel unwelcome and I want them to be fired?
Of course! I certainly don't hope that you'd be fired for that (nor do I think you would be in most places). You seem to be presupposing the reaction management will have, which makes me think that your problem isn't so much with cancelling, but with feeling unable to express your opinion that "white privilege" is a bad thing.
Now, you might not get what you want, which in this situation is perhaps the people who use the term white privilege to be fired. But that's because management doesn't find your complaint valid, which has been my point from the beginning. Your company has agency and culture that drives their decisions irrespective of what people say to them.
> If my idea to improve the workplace is to have all those who express feminist ideas to be silenced or fired, because as a male they make me "feel unwelcome", what do you think? Is it all fine?
My entire point, this entire time, is that individuals should be free to express the opinions they want, and companies should be able to act on those opinions by choosing to associate with who and how they want based on the company's values. If you can find a company that agrees with you and manages to avoid breaking employment law, more power to you. However if your company disagrees with the opinions you express, ultimately it is up to them what action they take. They are free to disassociate with you.
That's all cancelling is: individuals and groups choosing to use exercise their agency and freedom of association. And I support that. So yes, complain about whatever you want. But don't take umbrage when people choose, of their own volition, to disagree with you and to refuse to associate with you.
Ultimately, this is an issue that oppressed people have dealt with forever: that actions have consequences. It's great that society is getting to the point where everyone can face consequences for doing bad things.
> Of course one can ignore the political pressure, media articles, and very vocal minority who asked for him to be fired. But then it becomes a different story.
No one needs to ignore it. You however are ignoring everything else. That's my point.
I agree with what your saying in general, but this isn't really an instance of someone being taken down. I read it like when one of us normal folks deletes our social media accounts for mental health. Twitter seems like a toxic environment, but it didn't silence him any more than getting sick of the constant weird reality distortion on facebook made me delete mine. It just got toxic enough for him to decide to shut up.
The distinction matters because sometimes people not aligned with the zeitgeist do face serious consequences like being fired, receiving violent threats, etc.
> Especially if, in the end, we each still stand alone in the voting booth and pull the lever based on how we've been made to feel.
For a lot of people, the only way they can safely say that the social movement has gone too far, is by voting for Trump. If they disagree in public, the mob will come for them. However, in the privacy of the voting booth, they will pull the lever for the one person they know these people hate, not because they love Trump, but because it is their one way to protest against the mob.
I agree with your gist but let's not pretend that Yann LeCun has been "taken down." He is still tremendously wealthy, respected, influential, etc... it looks like the only change is that he's not on Twitter.
If an individual retains their personal wealth and lifestyle, but is unable to speak publicly - at least on one major platform, perhaps on many or all - then I don’t think it’s a justification of mob culture to say “well, he has a shiny car.” It doesn’t make the mob any less hurtful or hateful, and it doesn’t somehow redress the harm that’s being done to debate and free speech, which in the long run will probably have very damaging consequences not so much for him as for the mob.
Calling this his choice is an odd word selection, if someone is under extortion or blackmail do they give in by choice? If option A is horrible and one has to chose option B, is it fine and dandy because it was made by choice?
Maybe taken down isn't the right term, as if he's some great public figure. Whatever you wish to call it, forced to resign, or voluntarily said "I've had enough of this", etc.
I think the point still stands that if offering reasonable observations under such circumstances leads to this happening to such a person, it could equally well be you or I, who likely have little to no chance of being defended or heard in public.
Can anyone provide a simple explanation or example that illustrates why Yann was wrong? I’m highly predisposed to take his side in this, but I wonder if I could be missing something.
This is fundamentally a forest/trees type of situation. LeCun sees the issue with this one model, says, "If this had been trained on a different sample it wouldn't have this issue," and stops his train of reasoning there. The problem he is seeing is the mis-trained model, and nothing more.
But the problem is larger than this single model, because this issue (or similar ones) are pervasive in the fields in which AI are being employed. If a neural net is helping a court hand down sentences, it is going to be trained on historical sentencing data, and will in turn reflect the biases present in that data. If you are still only seeing the one tree, you say "well we must correct for the historical bias," and absolve yourself of thinking of the larger problem. That forest problem is that we will always be feeding these algorithms biased inputs, unless we do the work to understand social biases and attempt to rectify them.
It's not the job of the AI researcher to solve "social biases" in every field, it's their job to build the AI. LeCun is right to focus on his task and correct it, not just start talking about structural issues to which he cannot have insight anyway.
PS "Do the work" is a creepy phrase that is popping up everywhere in SocJus.
I recommend describing the "work" that needs "doing" instead of just saying "the work" need be "done".
> It's not the job of the AI researcher to solve "social biases" in every field, it's their job to build the AI.
No, this is only technically correct but actually wrong. In the example, they did in fact fail to build the AI, as is their job. "Recognize white faces" is a lame research goal, "recognize human faces" is the real thing. So if somebody builds an AI system that fails on out-of-sample data, then says that had they tried to do it properly they would have succeeded, that's a pretty lame excuse for a poor AI system. They didn't even do their job in the narrow sense that you're using, you don't even need to consider the "social biases" or whatever, it's just a system that didn't work. In fact, many years into this research program, "focussing on his task and correcting it" (working on all human faces) is still not done given the current performance of these systems, but they are quite sure it can be done if they tried.
> No, this is only technically correct but actually wrong. In the example, they did in fact fail to build the AI, as is their job. "Recognize white faces" is a lame research goal, "recognize human faces" is the real thing.
The researcher's job is to make progress towards the research goal. Progress that doesn't solve the problem is still progress. Nobody's saying that the PULSE authors have solved their research goal of generating white faces. It's why papers have a "discussion and future work" section, which touched on the issue in the first version of their paper. It's why in their revised paper, they added a full discussion of bias and the issues in the current model. They did their job, but didn't solve the whole problem, which is too high a bar for any researcher. Science is incremental.
That's how I interpreted LeCun's distinction between the researcher's job and the product builder's job.
Imo these are researchers. Their job was to validate their algorithm for doing generative super-resolution on a dataset, they chose the largest and most well-known dataset, it worked reasonably well on their dataset. The model itself is not productive ready for at least the reason that the dataset is not representative. This is ubiquitous in ML papers, they validate their idea on not completely realistic but widely available datasets. The outcome is a piece of knowledge about the behavior on that dataset not a product.
The point that many are making is that this is a myopic view of research. Why is the generic goal to optimize against some particular dataset? That ends up being a narrow and unhelpful goal.
As someone put on Twitter : we should be rethinking the meta learning algorithm of the ML academic sphere, and a leader like Yann is the kind of person who should be spearheading that.
Proactively doing the things that the ethicists are calling for. Use Model cards for any model. Actively think about and discuss the limitations the model, intended uses, etc. That goes for everyone.
For someone like Yann specifically? Publicly state that the ML scene is optimizing in a myopic way, and invest in doing so less myopically. For example, I think the translation space has a clear goal and the right goal and is making strides in improving language models in many useful ways.
Ultimately, if there aren't ethical and useful ways to apply facial recognition, leaders should be steering people away from those research topics.
The current nightmare that is cyber security is caused by developers who do not understand that with great power comes great responsibility.
The culture of software “engineering” and development is fundamentally broken and based entirely on “its not my problem if someone else gets hurt, I just build things.”
There’s literally not a single other industry where this level of greed and willful neglect is acceptable.
Of course, this would be reflected in the AI community as well.
That shows a fundamental lack of understanding of the fields of software engineering, cyber security and the fields. Both terms are individually overloaded such that many issues aren't even within their domain let alone actually handled by developers! No matter how good the lock is it does no good when the key is left on /top of tbe doormat/ by the user.
Blaming greed of engineers for security problems? Seriously?! That is like railing against EMTs as being responsible for the Coronavirus because they wanted to get rich without working hard. It conflates so many different areas and roles that it isn't even coherent logic and is nonsensical.
This is exactly the attitude to which I’m referring.
If you don’t understand that engineering is the fundamental bedrock of IT and the lack of security application during the engineering SDLC is a consistent failure then I don’t know what to tell you except maybe to gain more experience in software engineering and read more about data breaches.
But for arguments sake let’s just stick to network, application, mobile and IOT security.
- Why is MFA not on by default?
- How many devs have prod credentials published to Github.com right now?
- How many unsecured IOT devices are there?
- Why is email security such a dumpster fire?
- How many companies have an SSDLC?
- How many companies require separation of duties and approvals before a dev publishes another AWS bucket or some other unprotected data store to the web?
Go ahead and blame business but they don’t know jack about software engineering. We determine which corners to cut as opposed to stating to business that security is just a part of doing engineering. We’re the one’s who decide and thus it’s our responsibility despite denials of people like yourself.
A civil engineer should be able to do that (or decline to do it if they can not) and that's why it is their job and not the geologist's job to build the bridge. Research != engineering.
I agree with everything you say but it might be worth considering the "social" impact and aspects of AI models that are touted to change the world and are being rapidly adapted to our everyday lives with or without our consent. I can understand that engineers are not trained to consider those aspects but leaving them entirely to their non-creator or external bureaucrats might also not be the best strategy since they hardly understand the systems as well as engineers do. I am not sure what is the best strategy for this co-existence.
If Bagger 288 tries to drive across a footbridge that's not really the engineer's fault.
Although while everyone intuitively understands the load bearing capacity of a footbridge, not many understand the capabilities of ML models. So perhaps better advertising the capabilities of AI would help inform decisions.
On the other hand, nothing an engineer or scientist can do will stop the Chinese government from using their technology to predict and suppress dissidents and minorities.
I'm not sure if we can or can't, but in this case this is not just general "china bad", but an actually pretty relevant example because of the CCP using computer vision advancements for the purpose of rounding up Uyghurs.
Here we're talking about AI biases though. Such as the possibility of an AI giving a bad score to a black loan applicant because of implicit biases in the training.
On the other hand, China is in a conflict with the Uighur population of Xinjiang. As far as I understand it, there are elements both of cultural clash (China doesn't like religion in general, and the Uighurs are muslim) and the Uighurs' reaction to an influx of ethnically chinese population in the region. Anyway, Uighurs engaged in terrorism: this BBC article seems pretty balanced and lists a number of terror attacks by Uighurs as well as the repressive actions by China: https://www.bbc.com/news/world-asia-china-26414014
In this context, I think that China might be using AI not to "round up" Uighurs, but as an intelligence measure to prevent more terror attacks. Similar to how the US intelligence is (I have no doubts about it) profiling muslims and middle eastern immigrants- not because it has anything against those groups per se, but because it has reasons to believe terrorists might hide in their ranks.
All possibly true I guess, or at least I'm not an expert and won't debate you on this. Still, the discussion above was more about the potential uses of invented tech, and in that regard it doesn't matter if it's only rumored to be used wrong and the rumors have something like 10% chance of being true. The ethical dilemma is the same. It's there from the point that you can imagine it realistically happening.
> in that regard it doesn't matter if it's only rumored to be used wrong and the rumors have something like 10% chance of being true
I agree, but then it's telling how in an abstract argument about the potential misuse of technology seems natural to throw in an offhand accusation against a specific country. I am pretty worried by how quickly China has become the new boogeyman- everyone thinks it's perfectly reasonable to display anger towards a huge country that only a few years ago was seen, despite its obvious issues and shortcomings, as successful and dynamic.
And I remember how it started: when a mainstream, trusted news outlet reported about the existence of "spy chips" in hardware sold by Chinese companies to the US "according to extensive interviews with government and corporate sources". Which later turned out to be fake news. It gives pause for thought.
> China might be using AI not to "round up" Uighurs, but as an intelligence measure to prevent more terror attacks.
That is exactly what I claimed they were potentially using it for.
I acknowledge that I am very much biased against China as I live in a country within it's sphere of influence and have friends in HK.
But in any discussion of using AI for unethical purposes, China is the ur-example, as Nazi Germany is to facism: an authoritarian government with a history of tech surveillance, censorship and media control, and minority oppression, and the tech to back it up.
If they don't like that, maybe they should stop using technology to oppress their own citizens.
Yes LeCun is right to focus on his tasks as a researcher in research labs, papers, conferences. However, he is also a leader of the field when speaking to the outside world, which is what happens on Twitter in heated arguments.
I think he has some responsibility to at least acknowledge the complexity of this issue in such cases. Not speaking to the public is also a option if he doesn't like the leader role, so him deleting twitter is a completely ok thing in my eyes.
All sides agree there is an issue around fair implementations of ML algos in the real world. There are two ways this can happen:
1) The model uses biased data when it could have used unbiased data.
2) The model uses biased data when no unbiased data exists and is incapable of correcting for these discrepancies.
The first case is most clearly and engineering/implementation issue, the second is obviously not. Biased data is a known failure case of ML, its the responsibility of researchers to design for it.
If one is working on AI dealing with facial recognition and is oblivious to the potential for bias, and the unethical applications of that technology, at this point in the game, I can only assume it's willful.
For a scientist doing a ML system to reconstruct pixelated faces, trained with white faces, why is he/she responsible for "insert larger problem" outside of her/his field?
Do he/she has to also care about the brutal Tantalum Wars because there are some in the electronics they use?
As far as I can see, ML is pretty new, and there is a lot of room for improvement. And I think people need to stop thinking the entire world is racist, or doesn't care or don't want to improve things. Changes take time, and won't happen today, or tomorrow or the next decade.
There is a constant, furious need to point fingers, followed by fear of telling or not telling just the right words on the right order to the right audience. This is not how we are going to solve anything.
>For a scientist doing a ML system to reconstruct pixelated faces, trained with white faces, why is he/she responsible for "insert larger problem" outside of her/his field?
Because we're all responsible for how the tools we built are used and what they enable, and how they affect society at large. That's what ethics is about, something which seems to be absent in the education of the modern citizen and in particular engineer or scientist, who is supposed to come to work, program things and not think too much about the impact their products have on the world.
You know I'm fine with that attitude if it's combined with actual humility. Often as not, however, these high profile guys are shooting off on Twitter about every issue under the sun. But when it comes to racism, suddenly they're just narrow technicians and it's nothing to do with them.
I don't know if it is about racism. I think it's normal to become defensive when accused. If I say "your ML model is garbage" I am sure they will (maybe, if I am high-profile too) provide an explanation on why or why not.
Sorry, I expressed it wrong. I wanted to say that when someone is accused of something, including of being a racist or something related to racism, it's normal to be defensive about it and take that posture.
I don't think the researcher was being defensive just because it was about racism.
I think the key issues are primarily related to race.
Many see these systems as a way to implement stop-and-frisk (quite racist outcomes) and worse across the country using AI/ML as cover. The higher error rate amongst darker skin people gives LE a new excuse to harass innocent historically disenfranchised people.
I expect this tech will be widely used long before the accuracy problem that affects ~60% of the global population is fixed.
Reconstructing pixelated white faces is a lame research goal. If they think it would work on all faces, make them show that: it's science and it's their job to demonstrate that their approach works. Why bring "larger problems" into this when the problem is a "small problem", they built a system and it doesn't quite work but they published it anyway.
By the way, ML is not nearly as new as you seem to think, and given the amount of resources poured into it by FAANG-type companies recently, even five years is a lot of time for ML nowadays.
> Reconstructing pixelated white faces is a lame research goal
I'm sorry but you can't know this. Maybe it was the real objective, or a first step for something bigger, or a drunk Saturday night project.
In any case is research, and is very valid. Unless you are the chief at their lab, I think you don't get to tell what to do or not to do and/or the scope of their job. Anything else is a conjecture.
> By the way, ML is not nearly as new as you seem to think
Well, the results are there, together with the polemic it generates TODAY about white and black faces. Tell me if ML and its adoption, generally accepted or not, is mature yet.
All input is biased. It is not our job as scientists to remove biases from a data set. It is our job to reduce bias and characterize bias, but it's impossible to have a data set that has zero bias. You cannot sample the whole universe.
This issue reminds me of the IQ testing and measurement problem of regression bias, which likewise arises from statistical artifacts from decades past (when tests were administered to much more narrow and self-selecting groups of subjects than today).
The means calculated using much less diverse series of historical cohorts are still being fed back in to present IQ calculations, which thus retain a persistent corresponding "echo" of the biases in favor of those early test results.
The echo is gradually fading, but some say a clean reboot of the baselines is the right way to resolve historical sample biases which still skew IQ testing away from an accurate modeling of the diversity in present test populations.
The whole thing is an illustration of what a mess Twitter is. Sorting out what LeCun's overall message is seems next to impossible given the patchwork of ten line messages from who knows who.
But I just wanted to say that in the instance of a neural net that grants bail or not, there's an issue beyond either biased data or the neural reproducing previously biased opinions.
The modern notion of fairness implies that the individual be judged based on their personal merits rather than things roughly correlated with their surface characteristics. Being black is correlated with being poor and being poor is correlated with being a criminal and various other bad behaviors. But that doesn't mean it's fair to punish a given individual, who is only responsible for themselves, for such surface characteristic.
Which is to say that if an AI crunches the numbers in a objective fashion with the aim to make decisions based on various correlations, that can fundamentally problematic regardless of the bias of the original data or people.
What do you mean? "If blacks were criminal at the rate predicted by their poverty rate, they would be much, much less criminal than they actually are" is a pure empirical observation.
The logical point of comparison would be the nonblack population, but the conclusion is just as true for the entire population.
This is because it's not just a poverty problem, but also a class problem, with a healthy dose of racism mixed in to keep the status quo.
Lower class people are excluded from many (and sometime most) parts of society through harassment, violence, ostracism, or even through legal means (India's untouchables are a good example, or Japan's burakumin). As a result, they stop identifying with anyone outside of their class, and not only don't care what violence and misfortune befalls the other class, but are much more willing to contribute to it, given a good enough opportunity.
When you blend poverty and classism, you have the perfect mix for despair, because 99% of people born to this state have literally no hope of ever escaping it, which makes the criminal route a MUCH more attractive option, since virtually every legal means is barred to you by your class, and the few that remain are such long shots that you'll probably waste your whole life chasing them if you did (unless you get lucky).
I lived in the USA for many years and saw it quite clearly: the rage, the despair, the hatred. And I really can't say I blame the downtrodden for their reaction to this kind of systemic injustice, especially since they are treated this way the moment someone sees the color of their skin (the Irish and Italians of old could at least LOOK like the upper class with enough effort). How can someone who is identified as low class on-sight be anything but angry and frustrated?
But why is criminality comparatively higher and lower between different racial groups of the same class?
Edit: I know HN guidelines say not to complain about moderation (I've been here for 13 years now), but it's ridiculous a very simple, on-topic question should be downmodded into oblivion on a site whose purpose is discussion. While I have karma to burn and will happily keep participating, I'm concerned that this trend will have a chilling effect on people who do not.
Economic class and social class are different things, although one can affect the other to a degree.
You observe similar criminality trends in African immigrants to Japan compared to the other immigrant groups (including Europeans and Americans), for example. The only exception is the Brazilians, who were brought in as low class workers, have a MUCH higher crime rate, have trouble renting outside of low-income areas, can't get high class jobs, and are generally treated like criminals on-sight.
Germany has a similar problem with their Turkish population.
And it's not restricted to skin color, either. Low class culture takes generations to clean out. I've had a number of dealings with shady descendants of American-Italian immigrants, many in high level financial market jobs. This is fallout from the exclusion of Italian immigrants from the higher classes in the early 20th century, which led to an undercurrent of criminality in their culture (not to mention their good positioning during alcohol prohibition, gambling prohibition, and later drug prohibition).
Cultural currents take multiple generations to steer, which leaves one with many chickens coming home to roost after treating a class of people like shit for so long.
> This is fallout from the exclusion of Italian immigrants from the higher classes in the early 20th century
I am Italian and I can assure you that those who remained in Italy are not different. Large part of the Italian immigration to the US comes from the south, where the different types of mafia are only the official and extreme degree of a widespread culture of corruption, shady dealings and reciprocal abuse.
Social status/class and your economic status heavily overlap, but are not the same. The question gets partly answered just by specifying economic status/class vs just class.
The court example was good. Trying to make a list of possible domains where social bias is a factor:
- court risk assessments
- loan risk assessment
- job application pre-screening
- law enforcement face recognition
- medical scan interpretation
...
So, in essence: justice, banking, jobs, policing and health care. Anything else? Seems like social bias can only affect a small proportion of the ML application domains.
The classic (multi-decade problem repeated again, and again, and again...) is face detection. Face detection models continue to get trained by primarily white developers/researches who continue to use primarily white people as their training set. And then they're shocked when their model performs terribly for everyone who isn't of European decent.
This case is so pervasive that it gets taught in basically every university that teaches ML - and yet those students go on to repeat the exact problem they were educated against.
> Face detection models continue to get trained by primarily white developers/researches who continue to use primarily white people as their training set.
I guess you didn't watch Asian developments in face recognition closely in the last few years.
Assume you want to train an AI to recognize shops or buildings for example for a Google car.
Well if you do it in the US with skyscraper, in Europe, in Africa, Middle-East or Asia, you will get completely different results and biases.
Also, I don't see how anyone has the resources to compensate such a social bias, unless they plan to do a shooting trip in hard to access locations and try to justify such as "Seriously officer, the reason I take all those photos is to make sure my machine learning model is unbiaised so I don't classify shops in your country as shacks."
And if we forget human activity, even looking at nature, the fauna and flora are different between areas their color, how sparse they are, etc.
Last example, applying sparsity to human activity, what is actually a town in some country might be classified as a settlement for example due to bias.
In this case, ironically, it seems that an AI could incorrectly depixelate faces of suspects caught by surveillance cameras to white instead of black. Which would lead to a bias against white people.
PULSE is published as an art project, not suitable for face recognition or upscaling. It can only generate 'imaginary faces'.
They didn't even train the GAN they were using, it was borrowed from another paper. They probably used StyleGAN because it was a nice high quality generator and they invented a novel way to use GANs so they needed a toy model to showcase their algo.
Agreed. I was only pointing out that even bias is a matter of bias: if white faces had been depixelated to black faces instead of the other way around, the authors could have still been accused of a racist bias (because of the hypothetical scenario I've described above).
I think Yann and his critics are focusing on different questions. IIUC Yann is looking at the narrow engineering perspective of how to build an fair ML system. Gebru et al seem to be talking about a much more abstract, societal trend-oriented, "let's think about if we should not just if we can" type of question, though it's still pretty fuzzy to me. Anyway, the epistemological standards of those questions are basically incompatible (incommensurable?), so discussion is impossible unless they agree on which one they're trying to answer. I'm starting to notice this kind of thing a lot.
One, if datasets are biased --- if you build your system to only work on white males --- then it may have suboptimal results for other groups. This is a common problem: you use your company's faces, or college students enrolling in data gathering exercises, etc, who are not representative of the population at large. We can fix this by being careful about dataset bias.
But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets? If you train something to predict, say, odds to default on a loan-- will it figure out things that correlate to race and be making mostly racial decisions? Different races do have different default rates, but we've decided as a society that it is unfair to use race to determine an individual's probability of default. But if we choose things that are correlates of both default rates and race, when is that fair and when is it just veiled racism (redlining)? What things are measuring a causal relationship and what things are just racism in disguise?
This second problem is much worse with ML, because we have the ability to accept a whole bunch more things into our models and explaining the rationale of why decision are made is much harder.
And of course, the first problem-- both bias in datasets during use and biases during research and development -- makes the second problem worse. It can't even necessarily be addressed by broadening the dataset and retraining: if, in this case, you do your research and training with just white faces, and report positive results, it may not generalize to work as well for everyone with a broader dataset.
Assuming good faith, I really think there is a gap in understanding why an "objective" viewpoint can be seen as blinkered or prejudiced.
If you're looking at things that correlate, you've got a lot of choices. Saying that your data set is representative, your algorithms are correct, doesn't mean your choices of what correlations to use are objectively or morally correct.
Suppose that you have a system that accurately tells you men have higher car insurance claims on average than women.
The same system might also be able to tell you people with low credit scores have higher claims on average than people with high scores.
Is it right or ok to use the first correlation and ignore the second? Or vice versa? Or maybe both of them are unacceptable? There's nothing objectively inherent in the correlations that tells you it's ok to charge men who are good credit risks the same as men who are bad credit risks. Or optimal economically! Those are independent and difficult questions.
This is tangential to the original thread; the point isn't that the world is biased. Obama's face exists in the world. The issue is that the model is biased in a way that the world isn't.
But it's completely on point, IMO: even if the model is biased against minorities in a way that the world actually is, it can be what we consider ethnically unfair.
You're worse off lending in traditionally black neighborhoods; the question is, is it ethical to make a map of all of these traditionally black neighborhoods and refuse to lend there? A model that takes this into account would be biased in the same way the actual world is, but many would consider it deeply racist and unfair.
I feel like you've missed at least half of what I was trying to communicate, because you're still presenting arbitrary discrimination as reality-based.
Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the true risks of individuals within the group. Because it's suboptimal, it can't be called an unique, objective reflection of the world.
All correlations are inherently imperfect and therefore unfair; if they weren't you'd have an definite causal relationship.
When you use one correlation, or set of correlations, maybe you have completely missed something that would be much better, not just from a moral perspective, but a business perspective.
If you discriminate based on gender, and haven't considered age (hypothetically it being legal), maybe the latter would be a much better proxy for the real causal factors. Your gender model can be correct in itself, and yet from a completely amoral perspective be terrible and uncompetitive. The model is not "the way the world is" just because it's technically correct in isolation.
> Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the people within the group who are better or worse risks.
No model is perfect. Odds are you can better predict risk by (directly or indirectly) attaching bonuses or penalties to given races, all else being equal.
e.g. Say, irrespective of all other measured quantities, people of race A are more likely to default because of other systemic racism against them. As a lender, it'd be completely rational to consider this and make "better" choices. And if you're not allowed to measure race, it'd be completely rational to find other variables that don't have an obvious causal relationship to credit risk, but predict race and thus have some information about credit risk. This is the exact kind of thing ML does.
Then, in turn, this becomes self-reinforcing. Because other institutions discriminate against race A, the risks going forward of dealing with race A increase...
One possible approach is to make a model that besides its intended prediction also predicts race, and penalise its ability to predict race with probability larger than random. If the model has representations that can't predict race, then it will have to use non-biased features to make its predictions. But I guess in many situations this would degrade the accuracy and it will not be welcome from a business point of view.
How would one do this? If you have a ML model with outputs classifying credit risk, and outputs classifying race, it's easy to learn coefficients that are bad at classifying race but still take race into account in classifying credit risk.
That is, what's your loss function and how does it prevent race from being considered in the credit decision?
It's a setup that is somewhat similar to GANs (and even closer to a related method called Fader Networks):
- a first network take the input data and return a representation A (like an embedding vector): let's called it the "censor network"
- a second network take this embedding A as input and is trained to predict the class that should be censored (for example the gender of a person) : the "discriminator network"
- a third network take the same embedding A as input and is trained to predict the real task of interest (for example the probability of credit default) : the "predictor network"
The idea is that, by training the censor to make the discriminator fail (predict the wrong class) while making the predictor work, it will force the censor to learn a transformation of the input data that keeps the task related information in the embedding A, but removes the information correlated to the "censored class" (and that could be used to discriminate).
Here's a reference about this kind of methods, but it's still an active domain of study in ML and there are many papers that followed this one: https://arxiv.org/pdf/1801.07593.pdf
It sounds like you're looking at things purely from the point of view of getting the correct average for a group. But whether or not you get the correct average doesn't tell you if you're using a good enough or the best available model completely apart from fairness or justice.
If you do some type of testing and you know 5% of the tests should come back positive, is there a difference between reporting 5% at random and actually doing the tests? Of course!
> It sounds like you're looking at things purely from the point of view of getting the correct average for a group
No, I'm looking at things from the point of view of making a model that fits well to the original dataset and then is verified in the actual accuracy it makes over time.
If adding race-- or inferring race-- makes the model substantially better in predicting outcomes, is it right to do so? Credit default risk is correlated to race, even controlling for other variables. Hence, using race would help you make more accurate predictions.
When you say "better", better than what? How can your model tell you that some other data would not work better? The fact that you are looking at a correlation tells you that it's one of many ways to infer what you want to determine, and that it's imperfect. This is logically certain if we agree that race is not causal. So the only question is how much better is another model with different inputs.
I'd argue that curve fitting isn't modeling. And when inserting a fitted curve into a feedback system you're very likely to just perpetuate the problem you're looking to eliminate.
This entire discussion is about ML models, which can often be fairly described as very fancy curve fitting.
> And when inserting a fitted curve into a feedback system you're very likely to just perpetuate the problem you're looking to eliminate.
This is not a problem for the individual credit issuer-- they're not looking to eliminate the problem. They've avoided some credit risk by taking race into account. They've improved their expected value, even if society is stuck with the cost of the problem getting worse.
Sure, but "there are racial disparities in this domain, and therefore in the data" and "there aren't racial disparities in this domain, but there are in the data" are very different problems. The latter is the topic of discussion and is much less ambiguous than the former.
There's also "there are no racial disparities in the training data, but the system still shows a response with racial disparity."
ML has problems with all of these at times. The particular case here with the Obama picture is a particularly vivid illustration of one that can be used to bring attention to the overall problems.
It is not legal to make such a map. In practice, production actuarial systems for loans need to be interpretable and auditable, so this argument is moot.
In practice, multiple advertisers have been in trouble for breaking this class of rule in the past 2 years. That's not some long-forgotten problem, it's current.
If neural nets provide a way maintain plausible deniability while breaking the rules, it's going to get used.
You’re missing the broader picture. This is not a problem with data, AI, algorithms or whatever. This is simply a mathematical issue that is fundamentally unsolvable.
Imagine you have two groups of people that have different means and distributions of “input” (could be intelligence, wealth, interests, ...), and this input is somehow correlated with some output (insurance rate, criminality, university acceptance rate). Both input and output are one-dimensional.
Then you only have two choices. (1) You “unify” the distribution and use it as joint input, resulting in one group (the one with the higher mean) getting “more” - many people think this is unfair. (2) Or you can subtract the means first and so make the distributions as equal as possible, which can result in a person from the “more” group having worse outcome than an equivalent person from the “less” group (if the means are close together and the distributions are wide, there will be many such people) - many people would consider this unfair.
There are plenty of examples, current and historical, of society picking one or the other, for different groups and topics: wealth/success/income (old vs young, Jews&Asians vs other Americans, men vs women, whites vs blacks), university acceptance (Jews used to be discriminated against (look up “Jewish problems”), now Asians are, men vs women [1], “positive” discrimination of blacks) - just a few. In many examples historically, we consider (1) fundamentally better than (2) - e.g. discrimination against Jews at universities - but it seems it takes the society a long time to reach this conclusion in each instance.
Approach #1 is to just go to maximize expected value, using all variables, including race.
Approach #2 is to use race to adjust the distributions as you suggest. Call this "affirmative action."
The inbetween approach #1.5, that we usually follow, is to make a value judgment about whether any individual item is OK to use.
Income? Of course it's reasonable to use income to determine whether to lend-- even if it's correlated with race. Race? No, that seems wrong. Location? Sure--- wait, you've drawn a box around all the black neighborhoods, nope! Buys certain products? Sure-- wait, you don't mean "buys products that blacks stereotypically like," do you?
#1.5 is already iffy, but becomes completely untenable once model complexity gets high and we use approaches that do not do well at explaining their decision: it basically becomes #1.
A major problem with #1 is it becomes self-perpetuating and reinforcing: if everyone is going to be biased against you, you're going to do worse, and hence the models are validated/more models are created with these assumptions. Everyone using #1 may improve their own expected value at the expense of a worse outcome for society as a whole.
> A major problem with #1 is it becomes self-perpetuating and reinforcing: if everyone is going to be biased against you, you're going to do worse, and hence the models are validated/more models are created with these assumptions. Everyone using #1 may improve their own expected value at the expense of a worse outcome for society as a whole.
Do you have any proof of that? The world is improving on all levels, even the income gap between blacks and whites has been narrowing [1].
We're not talking about 100% segregated populations like during slavery or patriarchy, the means between the groups are very close and there's a huge overlap between the distributions. It's obvious that the vast majority of outcome isn't determined by race (otherwise you couldn't have white homeless addicts and black presidents) or any such "group" characteristic but instead by "individual" characteristics (skill, intelligence, drive, effort, hereditary wealth, chance, ...), so the evidence for this "self-perpetuating" unfairness is very weak.
I'm completely overwhelmed by your evidence of median black income moving from 59% of whites in 1970 to 65% in 2016, and appearing to not change substantively in the last 15 years of the dataset. If the overall trend continues, we'll reach parity in another 269 years.
And it hardly has been happening in a world where considering race and proxies of race is considered "OK".
This is an absolutely comical piece of evidence you advance to try and justify your view.
Pricing risk accurately is the lenders' job and we certainly don't want them to stop doing it. If society as a whole wants these loans to happen, society as a whole should underwrite these loans or subsidize the interest.
I've thought about this, but it risks creating an incentive for lenders to exaggerate the risk of minority lending in order to capture larger subsidized interest amounts or externalize more underwriting risk.
I don't really understand what it means to not use race as a criteria. If we're comparing Italians and Irish, and Irish are mostly left handed and Italians mostly right, and we know that right handers are more likely to pay back their loans, then presumably my race/ethnicity blind program will favor right handers over left handers, but the consequence will be favoring Italians over Irish.
When more variables come up and get more complicated it seems like it would be impossible to say "You're based on race" versus "You're based on a multitude of factors, several of which correlate to race, and consequently you give more or better loans to race X over Y."
>But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets?
Yeah this is tough, answering the question of the odds a person may default on a loan will probably reflect historical bias. In fact I'd argue that it should show this if the model is accurate.
The moral issue is what do you do with that information? Do you use that knowledge to improve the outcomes of the disenfranchised or do you reinforce the bias based on the information you have?
That is difficult to do because there are many things that correlate with race. So even if you remove race explicitly, things like neighborhoods, level of education and such end up being proxies. Removing race from the model is a very hard problem.
One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't.
I guess the question is why would that be germane to issuing credit? Is it correlated to something that indicated actual credit risk? If it makes erroneous assumptions correlations, would it not behoove them to eliminate the contamination?
Sure: some of your purchase history definitely indicates behaviors that indicate you're a credit risk. Some of your purchase history indicates you're likely black (and if you're black, your credit risk is, empirically, a little higher-- even if it is not fair to not issue credit to people because they're black).
ML will figure out both but can't explain the hidden variables: it'll figure out behaviors that directly indicate higher risk, and behaviors that indicate you're a minority that tends to be a higher risk.
By the way many methods are excellent at explaining their decisions. The favorite is decision trees which break the decision up into discrete decisions for individual features. E.g. a person was put in class 1 because their income was above X, their savings was above Y, etc. Other "shallow" methods like regression are fairly easy to interpret as well.
It is mainly deep learning for which this is difficult. Which I'm sure is the method in the twitter discussion, but that is a different kind of problem where interpretability doesn't have a clear use.
Basing your entire decision on something as simple as the applicant's income will presumably produce outcomes that differ with race. Is that what you mean by racist? Or do you mean the system identifies some individuals as high credit risk when they aren't and may actually lose money versus a more colorblind algorithm?
Many people would say that identifying people who spend a lot of money at bars or casinos as a credit risk isn't racist, even if it happens to pick up more minorities. The mechanism for the credit risk and behavior seems tightly correlated.
Many people would also say that identifying people who spend some money at clothing retailers that market to minorities as a credit risk is racist. Here, the relationship seems like a hidden way to spot minorities, who happen to be more of a credit risk.
When banks drew red lines around all the minority neighborhoods and didn't lend to people there, because they couldn't consider race anymore, most thought this behavior unacceptable, even though it did genuinely reduce credit risk for banks. ML can't explain its rationale, and very often does the exact same thing inside an opaque box.
Presumably black people who are great loan customers would be incorrectly refused by a system based on fashion or addresses alone, so your answer seems to be that yes, racist methods actually lose the bank money.
My tendency is to think that since it is optimized to maximize profit, the more features about a person we get in the data, the less "racist" the system becomes under this definition. Yes it's more possible to "hide" racism in a complex method using tons of features, but increasingly less likely it would happen. If we can use income and debt and whatever other things to make either a good classifier that ignores race, or an inferior one that sneakily identifies race and bases the decision on that, mathematical optimization of the model should result in the former. ML can be trusted more than humans in this situation, not less.
Of course there's still the issue of the data being biased, which is where it all started.
> Presumably black people who are great loan customers would be incorrectly refused by a system based on fashion or addresses alone,
Strawman. The question is whether predictions are improved by using race (or direct proxies for race), not whether it'd be wise to use proxies for race alone.
Loan default rates are higher for disadvantaged minorities, even after controlling for many, many other variables (income, neighborhood, level of education, etc). Therefore, using race (or inferring race) improves prediction quality, but is ethically dubious.
Can we skip this kind of snippy arguing please? Anyway I take it you agree the statement is true but don't want to.
So now you're saying those great minority loan customers are impossible to identify? I think you just need to figure out what information is still missing. What's the effect size at this point anyway?
If you don't start off by willfully mischaracterizing your opponent's argument in order to be able to more easily refute it, I think you'll find that they accuse you of this less.
I think you're just willfully missing the point.
Ideally, you'd just lend money to the people who will pay you back. Unfortunately, we can't predict this perfectly. Adding race, or proxies for race, to the things you consider improves your predictions somewhat.
Nope it was a completely honest question and what I considered a constructive line of thought. You should assume good faith. I asked about two competing definitions of what is racist and you seemed to prefer one over the other.
And what have I claimed to "easily refute" exactly? More like I ran with the definition, and considered how to address the problem as stated. I said more features were needed, I didn't say enough features were currently used. You keep pointing to a dichotomy between perfect and flawed, while I was talking about relative improvements. There isn't even necessarily a disagreement there.
I sez, "other variables + race makes better predictions of default risk than other variables alone". U sez, "LOL -only- using race will cost banks money" I sez, "???"
I wonder what a court would rule if you went through your algorithm line by line and demonstrated that is was only designed to maximize profit, and the fact that it was racist was only incidental.
I'm guessing it's illegal to discriminate based on race, gender or ability no matter how much it extra it costs a business, whether its deliberate or not.
Well in places like NYC the laws are strict regarding rental housing. Landlords need to establish consistent requirements then accept the first applicant that fits them. Typically they require the income to rent ratio is greater than three or something like that, which undoubtedly has a disparate effect when it comes to people of different races given how high the rents are and how incomes differ. Are you calling this incidental racism?
The issue is that "fashion for minorities" is correlated with being a minority which is correlated with poverty which is correlated with high credit risk.
This makes "buys fashion for minorities" quite an accurate predictor of high credit risk, but also a profoundly racist one.
Is it too entangled to say disregard anything that leads to a path that makes inferences based on demographics? Although I suppose age is also tough to avoid in assigning risk.
Almost anything you'd use is correlated to demographics somehow or another. We tend to think certain systems are "OK" and not too racist, because they have a well-articulated rationale independent of demographics. But ML is not very good at explaining rationale for decisions, so we don't have the benefit of that argument and it's difficult to even empirically understand how biased the system is.
Capital One has admitted doing the same, as well as a number of other lenders.
There's been analyses done since, that show that when controlled for all other variables in credit reports, credit issuers issue less credit to blacks.. e.g. Cohen-Cole, 2011, Credit Card Redlining, Review of Economics and Statistics
I said "One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't"
You asked for a source.
I provided a source of the same. Yes, things like purchase histories are used as proxies for race and used to deny credit to mostly people of color.
Then you go here:
> Credit Card applications do not ask for race.
which was never asserted.
> I suspect it’s more likely based on zip code or similar.
As for the latter, no, it’s not a proxy for race. It might be highly correlated with race, but that is very different. I don’t know why everyone does these gymnastics to find a racist angle to everything, but it’s nonsense and it’s very harmful to spread that narrative.
Nobody is searching for black zipcodes and using that as an input to deny credit. The goal is to increase revenue by giving as much credit as possible with the least risk possible, so it wouldn’t even make sense.
Scoring based on where you shop is still behavioral analysis. I think it’s a bad policy, but it isn’t targeting people of color. Your zip code affects insurance rates too, and that’s based on claims. It doesn’t cost more to insure your car in south side chicago than beverly hills because there are black people there, it costs more because there are more claims. The same effect is seen in “white” zip codes that are in areas with severe winters.
> I don’t know why everyone does these gymnastics to find a racist angle to everything, but it’s nonsense and it’s very harmful to spread that narrative.
Black people default on loans more, adjusted for income, education status, etc. But we've decided it's socially un-okay to ask people if they're black and adjust how much we're willing to loan based on the answer.
But instead, we measure all kinds of ancillary things, many of which are highly correlated with being black and not obviously correlated to ability to repay, and dump them into a model, where we come up with weights that make basically the same decisions as if we'd asked if they were black. Is this is fundamentally better somehow?
Mind you, I don't have a wonderful answer as to what to do: it's important to accurately price credit risk. But it's also important for society to treat minorities equitably, especially when inequitable treatment might reinforce the very problems/reasons why some minorities are worse credit risks.
It might be better to explicitly have the "is African American?" question in the model, because then regulators, policymakers, the public could perhaps eventually know the exact contribution of this factor... while approaching it obliquely makes the effect far less clear.
It's pretty hard to make the system "blind" to race. If you blind a system predicting recidivism to race, but it is able to see a lot of things that are correlates of race... it can end up being racist anyways.
E.g. for a non-ML example -- redlining is theoretically "blind" to race, but makes extremely racist decisions.
It's really, really hard to erase biases that are deeply systemic.
For example, from the outset, would you object to an AI that made decisions on how harshly to sentence someone based on age & number of prior crimes?
What about hiring or admitting people to college based on the results of an IQ test?
Yeah, those end up being badly biased, this first has been studied and the second is the reason for dropping SAT/ACT scores--every mental ability test correlates with IQ.
The more interesting thing, IMHO, is that the opposite is also helpful. For example, if you help all poor people equally, you help even the playing field by disproportionately helping out all disadvantaged groups.
Application of law should always have room for judicial discretion. Mandatory minimum sentences should not exist, but then you get outrage over some light sentence here people get all riled over about and want to impeach judges and all.
Education is different. IQ shouldn’t be relevant but competency and aptitude should and in addition we should recognize some people are better off going to vocational school where they may do better economically during their lifetimes.
The first one was intentionally minimalist in order to see how little it took for racial bias to be reconstructed.
For the second, any mental ability test correlates with IQ, so you end up with a different set of difficulties. There have been attempts to, e.g. correct for cultural bias in the tests, but these actually made the problems worse.
I'm not presently aware of anything that makes that situation better.
If that is the case they should notice they are talking past each other and reset the conversation... But I guess that is very hard/impossible once it gets personal. Thus I think the fact that it became personal is the main problem...
Twitter doesn’t help finding out the misunderstanding and do a reset, you have arguments/counter-arguments spread between thousands of messages with no overview of what is being said, what are the core ideas being discussed.
A variant of this is that both ppl are answering the same question but on different time scales. So one is pro (on a generational timeline) and the other is con (on a 4-year timeline).
Both might be right, for their respective time scales, which is why it's so hard for either one to back down.
Many many ways to talk past one another. We need a small book on techniques to short circuit these situations and salvage conversations.
And that's the right way to look at it. If you can't then you shouldn't. It's not enough to build stuff and then to say 'oh, we can just fix these serious problems' afterwards if you're not able to foresee / structurally address them from the get go. Because it clearly shows a much more shallow understanding of the problem space than mere engineering can fix.
Surely this comment reaches a little too far? We have to be brave enough to try new things even if they have some risk of unknown/unintended consequences.
No decision is completely safe and no system gets everything right in the first pass.
All we can ask for is a serious effort to understand risks sufficiently well to reasonably believe that the system is safe.
You can try 'new things' in the lab, until you are sure they work and are safe for general application. To field these AI solutions in lots of places where negative side effects are a serious possibility knowing that this is the case is irresponsible. Seeing this as a simple engineering problem with a simple fix-as-you-go attitude is not the right approach.
Just like 'temporary software' university research projects tend to eventually get fielded in some form or other. By not tackling this problem at the root we push the onus for correct implementation down to the people that field it who thought the academics had done a proper job in the first place with all their metrics pointing to 'success'. But success isn't a success when the dataset isn't representative and datasets used by academics as a rule are not representative or have built-in biases that are very hard to overcome by engineers aiming to field these solutions.
This has already led to numerous troubles in the applications for loans, job evaluation, criminal matters and so on.
> This has already led to numerous troubles in the applications for loans, job evaluation, criminal matters and so on.
Which I'm arguing about elsewhere... but demanding that university research about upscaling a face to be perfect to publish is a little much.
> . But success isn't a success when the dataset isn't representative and datasets used by academics as a rule are not representative
Yup, so, academics should never research this stuff or publish? That certainly is an ... interesting ... recipe for progress. And research progress is one of those things that could help us ultimately address some of these issues, because as the original argument makes clear: improving the quality of datasets is not enough.
LeCun - ML is biased when datasets are biased. But unlike deploying to real world problems, I don't see any ethical obligation to use "unbiased" datasets for pure research or tinkering with models.
Gebru - This is wrong, this is hurtful to marginalized people and you need to listen to them. Watch my tutorial for an explanation.
Headlines from tutorial (that Gebru didn't even link herself): The CV community is largely homogenous and has very few black people. Here's a bunch of startups that purport to use CV to predict IQ, hiring, etc.
Marginalized people don't work on these platforms and there's no legal vetting for fairness before these platforms are deployed. Facial analysis has the highest rate of inaccuracy (gender classification) on fair-skinned men (?) and dark-skinned women. Datasets are usually white/male. Most object detection models are biased towards Western concepts (e.g. marriage). Crash test dummies are representative of males, so women and children are overrepresented in car crash injuries. Nearest neighbor image search is a unfair because of automation bias and surveillance bias. China is using face detection for surveilling ethnic minorities. Amazon's face recognition sold to police had the same biases (greater difficulty distinguishing between black women).
Now, I largely agree with what Gebru said in the tutorial. So does LeCun, who explicitly agreed a number of times that biased datasets/models should never be used for deployed solutions.
But it's a huge leap in logic to then demand that every research dataset be "unbiased". It's like criticizing someone for using exclusively male Lego figures to storyboard a movie shoot, or if I attacked a Chinese researcher because they only used Chinese faces to train a generative model, and none the outputs looked anything like me.
That being said, I'm open to being convinced if she had made any effort to show/prove that "use of biased datasets in research" is correlated with "biased outcomes in real world production deployments". But she didn't, which is why her criticism of LeCun smacks of cheap point-scoring rather than genuine debate (a criticism I made of Twitter generally the last time this topic came up).
I became convinced of cheap point-scoring when LeCun invited Gebru out for a chat because he felt like twitter wasn’t the right medium and got a link with how to properly apologize back, but this rundown would have saved me a lot of time when I was trying to figure out wtf happened.
Do you believe that industry uses pre-trained model that researchers release?
Do you believe that industry uses pre-made datasets that researchers promote in their work?
Would yes to the above two question be sufficient to show "use of biased datasets in research" is correlated with "biased outcomes in real world production deployments"?
"What I believe" is completely irrelevant, because it has zero grounding in research or evidence. She's claiming that we should listen to her because she's an expert, but then provides no evidence or explanation between the two for a causal link.
I'm not saying she's wrong, I'm saying we don't know because she defaulted to the argument that "you should listen to minorities", not "here is the evidence".
What's more, every single example of injustice in her tutorial was an image recognition/classification problem - entirely different from the generative model that originally sparked the debate.
I don't think there's an argument that needs to be made; it's pretty clear to me that people use researcher-produced models in production all the time without regard for whether they're biased because it's the easiest solution. If you think there needs to be an argument made to justify that, that's fine, but I don't think it's valuable to assume that people come into a discussion without a basic understanding of the software engineering ecosystem.
And the point being made isn't "biased input data isn't responsible for a biased model", it's "you need to look one step further and ask why is the input data biased and how that impacts the world."
I want to point out that while I disagree with you, you've already engaged in far more actual debate than Gebru did on Twitter.
Like I said, LeCun (and myself) largely agree with most of Gebru's points. But when LeCun went to great lengths to defend/explain his position, Gebru then literally responded with "I don't have time for this". Even before that, she didn't even bother to link the presentation she referred to (which again, didn't even directly address any of the points that Yann was making!).
It's this complete lack of good-faith engagement that prompted LeCun to quit Twitter, not the underlying discussion on ethics itself. LeCun clearly feels that Twitter is not the place for reasonable discussion, and after this episode, I'm inclined to agree.
> If you think there needs to be an argument made to justify that, that's fine, but I don't think it's valuable to assume that people come into a discussion without a basic understanding of the software engineering ecosystem.
I'm not saying that people don't use off-the-shelf models. I'm saying that I don't know if forcing research datasets to be "unbiased" will make any difference to real-world injustice. I don't even know if any of the examples of bias she gave in her tutorial (HireQ/Microsoft/etc) could be ascribed to the use of pretrained models. She could be right. I don't know. You probably don't either.
Going beyond the empirical question, she'd also need to explicitly argue why responsibility should lie at the feet of the researcher, not the engineer. Gebru did neither, which is why I say it's a huge leap of logic.
That, fundamentally, is LeCun's position. He completely agrees that warning labels should be put on these kind of models that say "Model has been trained on biased data and unsuitable for use in real-world applications where racial fairness is expected". In fact, this is exactly what the authors did.
> And the point being made isn't "biased input data isn't responsible for a biased model", it's "you need to look one step further and ask why is the input data biased and how that impacts the world."
And I'd argue you need to account for the context in which your model is deployed. If I'm using StyleGAN to synthesize facial textures for a video game, biased datasets and models are desirable, not something to be eliminated. I'll use the appropriately biased model depending on whether I want to generate white faces, Chinese faces, or black faces.
It's the use case that dictates the risk, hence why LeCun (and I) believe it's the engineer's responsibility, not the researcher.
In fact, they did so after Timnit brought up her objection, and did so by taking advantage of Timnit's research (the model card they added is a direct result of research Timnit was involved with: https://arxiv.org/abs/1810.03993).
Which is precisely why LeCun - like myself - agrees with 95% of what Gebru has said in the past.
The issue at hand was her lack of good-faith engagement on Twitter and the subsequent pile-on from the mob. LeCun is quitting Twitter, he's not quitting ethical debates.
> Which is precisely why LeCun - like myself - agrees with 95% of what Gebru has said in the past.
Even now, it's not clear to me that this is the case. LeCun still hasn't actually acknowledged any of the broader ethical arguments Gebru made, either on twitter or on his followup posts on facebook.
In fact, he makes no references to her research anywhere (beyond the vaguest "I value the research you're doing" in his apology tweet). I found that rather suspicious, I still do.
Like, having read through the entire conversation, I have no confidence that Yann could explain any of Timnit's research if asked about it, even in broad strokes. That's really, really weird given everything that happened.
> Even now, it's not clear to me that this is the case. LeCun still hasn't actually acknowledged any of the broader ethical arguments Gebru made, either on twitter or on his followup posts on facebook.
Well, to be fair, she didn't provide Yann with any ethical arguments in this instance.
But being equally fair, you're right, I can't speak for Yann. I can only speak for myself, and I personally agree with most of what I've read of Gebru's work (though not all).
But the issue at hand wasn't the research itself - it's the way the dialogue was conducted on Twitter, and Gebru accounted for herself very poorly.
> Well, to be fair, she didn't provide Yann with any ethical arguments in this instance.
Yes and no. A lot of what I'm saying is directly from the tutorial she repeatedly suggested he watch. Because I took the time to watch it, because that's the reasonable thing to do when someone suggests that you aren't fully informed on a subject and suggests a resource to improve your understanding.
A major aspect of crypto research today is crypto UX and making crypto systems that are difficult to misuse. There are academics who actively work on these issues. They aren't the only academics obviously, but they exist.
Building ML systems that are difficult to misuse is underexplored, and Timnit is one of the relatively few researchers actively doing work in this area.
I'd call them both examples of applied cryptography research. I think these projects compare very, very closely to applied ML research:
They come out of industry research labs, are worked on by respected experts, usually involving some academics, ultimately you end up with an artifact beyond just a paper that is useful for something and improves upon the status quo.
I'm admittedly not a total expert, so I don't know how far down to the level of crypto "primitives" this kind of work goes, but I believe there is some effort to pick primitives that are difficult to "mess up" (think "bad primes") and I know tink actively prevents you from making bad choices in the cases where you are forced to make a choice.
Even more broadly, just consider any tptacek (who I should clarify is *not a researcher, lest he correct me) post on pgp/gpg email, or people like Matt Green (http://mattsmith.de/pdfs/DevelopersAreNotTheEnemy.pdf).
Edit: Some poking around also brought up this person: https://yaseminacar.de/, who has some interesting papers on similar subjects.
Misuse has two meaning: to use for a bad purpose (criminals using it to do bad things) or use it incorrectly (hold it wrong). I was using misuse in the "hold it wrong" sense, but I agree that there's ambiguity there.
That's not what I got from LeCun's comment. I read it more like:
LeCun - "ML is biased when datasets are biased. It's not the responsibility of researchers to ensure that ML is used responsibly in all cases, but the responsibility of the engineers of a particular implementation who need to use the correct models for the task at hand."
> "...I don't see any ethical obligation to use "unbiased" datasets for pure research or tinkering with models..."
I don't think his comment was addressing the larger ethical discussion at all. I didn't interpret it as a discussion of ethical responsibilities, rather a strictly technical, matter-of-fact statement about the nature of ML training.
Please don't interpret my comment as an attack on yours, it was more pointing out I interpreted his statement differently.
I guess I see your view and my view as two sides of the same coin - that “research ethics” is different from “application ethics”. I inferred that view from the following exchange:
Twitter user: “ML researchers need to be more careful selecting their data so that they don't encode biases like this.”
YLC: “Not so much ML researchers but ML engineers. The consequences of bias are considerably more dire in a deployed product than in an academic paper.”
Perhaps I’m wrong. That’s the whole problem with Twitter though - you can’t convey much nuance or sophistication in 140 characters.
your summary is not making a lot of sense either to be honest. well, at least the last part...
> I'm open to being convinced if she had made any effort to show/prove that "use of biased datasets in research" is correlated with "biased outcomes in real world production deployments".
what does that mean? is there anything that would auto-magically eliminate bias if it were introduced into research?
> what does that mean? is there anything that would auto-magically eliminate bias if it were introduced into research?
Let me rephrase. Yann is basically saying "bias is the engineer's responsibility, not the researcher's". Gebru (presumably) disagrees.
Now I might agree with Gebru if:
(a) she can show empirically that "researchers releasing biased datasets/models" is correlated with "real-world deployment of said datasets/models that leads to injustice"; and
(b) she can make a convincing argument why one person (a researcher) should be responsible for the actions of another (an engineer).
But she didn't address either these points on Twitter. She actually didn't bother to address anything on Twitter. Her whole argument was "You're wrong, I'm tired of explaining, you need to listen to minorities, I'm not going to engage".
That's not reasoned discussion or debate. It's posturing and point-scoring. The Twitter format only serves to encourage this type of interaction, so Yann basically gave up on the whole platform.
okay, so... you seem to understand where the other researcher is coming from and agree with most points. i am also going to assume that you read, or perhaps know, some of the sources cited numerous times on this page.
but because she did not explicitly state those on twitter, or because of the way she brought it up, we need to invalidate her whole argument?
> but because she did not explicitly state those on twitter, or because of the way she brought it up, we need to invalidate her whole argument?
No-one said anything that could be remotely interpreted as "her whole argument is invalid".
I'm sure he'd be more than happy to discuss with Gebru where he agrees and where he differs on his Facebook page or at a conference panel. I think he explicitly said this.
He's just decided that Twitter is not the platform for that kind of reasoned debate. Gebru's attitude in this instance - providing nothing more than "I'm tired of this, you need to listen to marginalized communities" - was the straw that broke the camel's back.
Because the points of disagreement are the reason she's upset, and the reason there is an argument in the first place.
Of course she's right about all the things that everyone agrees on. Everyone in the conversation is right about most points, if you break down their stance into a list of points.
It's not that the points of disagreement invalidate the correct points, it's that having a bunch of correct points doesn't really tell you much about the thesis.
Part of the problem is that "it's just the dataset" is being used as an excuse (witness the "just"). It doesn't really matter to possible victims of, say, the use of AI in law enforcement where exactly the problem originates.
I doubt that anybody was accusing the creators of that upsampling model to have intentionally tweaked the model to default to white people. In that sense, YLC is attacking a straw man.
Given the publicity such problems have gained in the community, one would expect publishers of any model to verify it doesn't fail with the most obvious examples. Not doing so is negligent at best.
If we lack datasets to train AI that doesn't spectacularly fail any test for racial biases, we lack datasets for anything that could be considered fit for use. At that point it doesn't matter if the model is flawed, or if it's "just" the available data. Such models shouldn't be published, and we should instead invest in either better data, or come up with better methods of training.
(And, as a minor point, his idea that Senegal is representative of "Africa" as a whole is also... let's say "unfortunate")
> Given the publicity such problems have gained in the community, one would expect publishers of any model to verify it doesn't fail with the most obvious examples. Not doing so is negligent at best.
Here's what the authors say about their own work:
> PULSE makes imaginary faces of people who do not exist, which should not be confused for real people. It will not help identify or reconstruct the original image.
Furthermore, this author appears to describe themselves as a more of an artist and hobbyist. This isn't someone making some kind of statement about ML research. This is someone playing with computerized art, and the entire social media tech mob dogpiles his work over what exactly?
The negligence here is on the part of everyone getting their hackles raised over nothing.
If they are in here I hope they don't misconstrue my linking this image. I think they explained themselves very well and I feel bad that they were thrust into the middle o f this controversy.
My point is that while diversity might help, you need a lot more than that to address this problem.
We can say bye-bye to our nice demos and pre-trained models after this debacle. Who's going to risk their ass just to be caught with some unknown bias?
> The negligence here is on the part of everyone getting their hackles raised over nothing.
To be clear no one (or at least no one of note) has their hackles raised over this specific dataset. PULSE is fine for what it is, and no one criticized PULSE for having these results.
It is however a great demonstration for laypeople about how ML models aren't magic and don't always do what you, as a human, would expect. This is true irrespective of the source of that unexpected behavior.
That said, I believe the disclaimer you mention was added only after the recent twitter discussion.
This all may be true but there’s an issue when the model produces blue eyes when presented with a blurred African American face. I think much of the controversy would be diffused had the authors addressed this directly and used it as a way to discuss how bias sneaks into our models of the world.
Philosophically I find ML’s tendency to reflect the biases we bring to it very revealing. In some ways it shows us what we’ve built, the underlying biases that we’d rather argue about and ignore. When an algorithm selects longer sentences for black men than white men, we rightly see that as racism. Some say, “use better data, we’ll then be objective!” But I wonder if a better initial reaction is, “wow, look how badly out system has failed that it would produce such a bad dataset.” Never mind that maybe it’s not actually possible to be objective and that’s the point. Math doesn’t lie, maybe when we make a racist model we’re failing to see the mirror it’s holding up for us.
Yes, they could have picked a classification model with social impact instead of a GAN. GANs are mostly toys for art and image augmentation. The bulk of models are supervised classification.
It's not the words, it's the idea that Senegal is representative of darker people and could produce the correct results. It's an indication that he still doesn't understand.
Using Senegal would make people look Senegalese, not "African."
> Not all black people are Africans or have African heritage. Far from it, in fact.
Exactly. That's the "rational" point: the mistake was training the data on anything but the target population. Nobody is contenting that it could have been trained on something else: it wasn't and that fact is the problem.
> Also: darker? Darker than what? Are you taking “white” as your baseline
Darker than the white people the algorithm turns most inputs into. Have you seen the results?
> It doesn't really matter to possible victims of, say, the use of AI in law enforcement where exactly the problem originates.
I cannot believe that so many people fall for this. Journalists, laypeople, even HN users. (OK not that surprised regarding journalists.)
The problem is not racial bias in AI policing. The problem is AI policing! Racial (and any other) bias is trivial to remove - just subtract the mean! But that doesn’t make predictive policing a good idea.
Imagine this:
> Hello, mister/lady, our 100% unbiased system has automatically determined that you are a potential future criminal. You are under arrest and sentenced to death.
(Edit: and same could be said for almost any situation where you have a bureaucrat making decisions about people’s lives, and you try to automate this decision with AI.)
It is "just the dataset" is a very valid excuse for not working. What if they trained the dataset on cats instead? Expecting it to work on humans instead .
What it isn't an excuse for are the goddamned negligent idiots who tried to use it in law enforcement without through testing. It would be akin to a surgeon dipping every sterile sharp unstruments in yogurt cultures and foregoing antibiotics before use to see if good bacteria makes infections less likely and could prevent antibiotic resistant bacteria. Even if the theory is valid and the goal worthwhile the needless risk taking shows a callous disregard for human life especially when done in an utterly halfassed way like that.
To me it seemed like Timnit Gebru was spoiling for any reason to fight, and decided to make a big deal out of a factually correct and hard-to-misinterpret statement.
I mean, he was clearly referring to the specific model.
Just browsing her Twitter made me cringe and I am a female PoC. Her response to every argument is "Ah yes the problem with the world is a PoC doing this and that" when the original argument hasn't even suggested that.
Recently seems in support of a black scholar who cried racism because her non peer-reviewed work that was only posted on arxiv wasn't cited in a lecture on GANs ...
Voicing these opinions would probably label me as racist in their book ironically.
> Voicing these opinions would probably label me as racist in their book ironically.
Don't worry, Twitter isn't real life. People have become experts at shouting down people who disagree with them on that platform, they aren't seeking proper arguments and make disingenuous attempts to present them as honest debates.
But fortunately what's popular on Twitter doesn't translate to the average population. Plenty of completely fringe ideas get 50-100k likes/retweets. It mostly just represents the voices of various super-niches living in bubbles.
Without taking a position on the original argument, the person involved seems to hold an influential position at Google and in the community, so I don't think this is necessarily a satisfactory response.
>>Don't worry, Twitter isn't real life.
For the most part yes, but President Trump's behavior on Twitter can have serious consequences in the real world. I think that the behavioral norms of social media are penetrating deeper and deeper into culture.
You seem to be taking "not real life" too literally. I'm not saying "nothing on Twitter affects real life". The point is that it's often a poor representation of real life.
Small but highly vocal groups can have a seemingly loud and powerful voice. Yet the results of polls and other public signals (even election outcomes) are frequent reminders that what is gospel on Twitter is often detached from 'real life'.
Using a president of a major country is a poor example in this context. But if anything Trump being one of the first major Twitter users strongly reinforces my point. Prior to election he tweeted plenty of things most mainstream US republicans wouldn't touch with a 10 foot poll. Let alone what an average American would say IRL (even right leaning ones).
Not to mention Twitter is a global platform so conversation around local politics can be heavily skewed by people not even in the country.
But otherwise I agree, it is infesting real life far more frequently these days. And it is worrying. Despite everything I said above, big corporations, the media, politicians, etc can't seem to make this distinction and take what is popular there as a direct reflection of the general public. And it creates a negative reinforcing spiral.
I'm glad you are saying this. I work at Google (and so does Timnit Gebru). Recently there has been a somewhat similar discussion on an internal mailing list. A very senior person at Google got involved in the discussion and backed Timnit up.
It made me feel very uncomfortable, because I wanted to say something like "the issues you are pointing out are very important, but I don't think you are right in this instance" but I was afraid doing so might jeopardize my career at the company.
As a POC, it is exhausting to have folks like Timnit spin everything into a problem devoid of a solution. I hope we take a scientific approach to problems and solve it rather than driving away folks who can help in solving the problem.
As a POC, it bothers me that someone from the "corporate(or educated) class" of POCs always coming in to criticize those POC who have a legitimate concern about some injustice they are seeing. In this case Timmits concern is entirely valid and does not have a solution currently other than do not use ML for some applications. We benefit from Timmit and others voicing these valid concerns. Engineers always want to base everything on the data to take themselves out of what is being asked of them. It is not always possible to reduce problems to "the data". POC in particular should not be quiet when it comes to some of the issues around the questionable use of ML in relation to race and issues that are ultimately surrounding race.
It makes sense once you realize she is only using this to get an executive position in some institute for removing racial biases in AI and be set for life.
I'm in the group which was upset over the insinuation that ML Researchers should not care about fairness or ethics as much as ML Engineers. The distinction between researcher and engineer should not be grounds to care or not care about safe and ethical model building. This is especially true when the difference between either role is highly arbitrary and varies by organization or field.
For the sake of ethical R&D, it's counter-productive to build a hierarchy of investment into the problem. Admittedly, the responsibility of end-results can differ, but the consensus that this ethical work is important should ideally be universal. That said, there should not be some ivory tower where you wash your hands of the ethical problems of your field.
I don't understand what Timnit and these other people are really even advocating. She says "diverse datasets are not enough". Structural problems can't be ignored etc.
Ok, so what is the solution then? Does she have a concrete set of steps or goals to address the problems she sees? Is there a list somewhere of things that would appease her, and in her mind make ML fair? Honestly interested.
Lacking an immediate solution to a complex problem does not mean these problems don’t exist. I’ve noticed often lately that people, when talking past each other, one person is saying, “We really need to consider the implications of problem X.” and the other person will imply “If there is no obvious solution, then no one should consider this a problem.”
When it comes to problems, particularly in complex subjects which aren’t yet well understood, and where there aren’t yet an overabundance of high caliber researchers, this comes across as dismissive of the problem.
This can be even more concerning if we know there are investors lined up who will happily sell something to the world and who will intentionally hide or minimize known ethical concerns. And then play dumb and shocked later when the very same problems manifest.
I see this dismissal of justifiable and real concerns an awful lot in conversations of all kinds lately.
And from what I’ve seen, no one here is anti-ML, no one on either side here is a luddite. But like so many conversations online, we should quit talking past each other and likely need to quit trying to paint people as if their concerns don’t have very real ethical implications which will, if left unaddressed, manifest in all kinds of negative ways throughout society.
Again, a lack of a neat and tidy solution doesn’t mean the problem doesn’t exist.
I can't speak for Timnit, but for me it would have sufficed to see Yann start with "I care about ethics and so should researchers and engineers." Instead it took so long to get there, so many "buts" and so many implications that researchers shouldn't be as invested, that I was disappointed.
As for Timnit and LeCun: a user above notes that the epistemological frameworks they're using to analyze this problem are not aligned. I found that comment pretty eye-opening, honestly.
In this case, the very existence of teams is an invention of those who managed to make a career as team leaders. By prefacing your statements with an explicit allegiance to a team, you're in fact endorsing whoever is presenting him/herself as the team leader.
This is purely a power game in which some individuals have managed to blackmail everyone else into recognising their role or being cancelled. Now and then someone prominent needs to be attacked to remind the others what they risk if they don't fall in step.
My experience in ML and computer science is that it cannot be assumed. Consider companies like Palantir and their Gotham system. Consider the facial recognition systems deployed in Detroit, which fail 96% of thee time. It especially cannot be assumed when you attempt to offload that responsibility to other people.
Computer science programs around the country promise their young, bright-eyed undergrads that they'll change the world. Very few of them teach them the ethics they'll need to do that in a thoughtful way.
The assumption that science and technology is inherently ethical has unfortunately led to dangerous ideas over the past 150 years. Some, like geographic determinism and eugenics, have directly led to the suffering of millions of people. I hope we tread carefully and take action when we see harmful models. [0]
> The assumption that science and technology is inherently ethical has unfortunately led to dangerous ideas over the past 150 years.
I think that the underlying assumption is that science and technology are inherently neutral (which isn't true) and that neutrality would be inherently ethical (which also isn't true).
If you're saying CS people need to read more humanities, I'm absolutely with you.
But I didn't say that science and technology are inherently ethical. I said that most people care about ethics. They might have different priorities or philosophy than you, but almost nobody commenting on a social issue is doing so because they want the immoral thing to happen. Right? So asking everyone to say "of course I care, of course" before everything they say is laborious.
The solution they seek is to stop pretending that research happens in a vacuum, to think about likely applications of the technologies we develop, and stop taking for granted the idea that the invisible hand of the market will determine the optimum application of the cool shit we make.
This is an extremely unpleasant position to take, if your point of view is empowered within the status quo. It is much extra work for no discernible benefit to the researcher.
If your point of view is subject to disproportionate suffering under the status quo, then reinforcing current practices by implicitly enshrining them in input datasets will make improving your situation even harder.
As an example, consider the case of public school funding. In a hypothetical system where school resources are provided proportionally based on student success, good schools will thrive and bad schools will get worse. If someone points out this isn't fixing the problem, you can reverse the proportions -- this will cause good schools to suffer while bad schools will get additional funding (disincentivizing student success). In cases like this, it's not enough to just have a purely abstract set of metrics on which to base resource allocation: it will always require actual investigation of why good schools produce good results and why students do poorly in specific schools.
This is sort of obvious, of course, but it isn't being translated into terms that some researchers can or will grasp. It's never enough to just tell someone to 'debias the dataset,' as determining that bias is a monumentally difficult challenge that people have failed to achieve for many generations. A key factor in fact is the propensity for this kind of research to get deployed, today, by people who are not experts in a given domain of investigation, with possibly disastrous results in policymaking. These tools are not abstractions that require a team of experts to translate from research paper to the real world; ML researchers put out results that you can shove into your nearest computer and run.
What Timnit and others are getting at is that it requires thoughtful and careful assessment to get real value out of this sort of research. Ideally, in Timnit's assessment, the researchers themselves would put effort into identifying possible calamities and put as much effort into mitigating them as they do into publicizing the work itself.
Yann LeCun and other researchers simply do not believe this is their responsibility; all they want to focus on is the mathematics themselves. I'm sympathetic to this position but I also very much do understand the opposition. One of my favorite movies from childhood, "Real Genius," deals with this sort of issue as the main plot line.
> it's not enough to just have a purely abstract set of metrics on which to base resource allocation: it will always require actual investigation of why good schools produce good results and why students do poorly in specific schools."
The problem here is that many stakeholders really really* want to reduce the required intervention to a mechanistic increase/decrease of some sort, never mind that different stakeholders want opposite interventions.
No matter how "intelligently" you go about it (whether the intelligence is natural or artificial), this desire to simply boil down policy making to reallocating resources or incentives/disincentives is fundamentally lazy. It reminds me of the mania for diversified conglomerates and corporate management of the firm-as-portfolio (ie. reducing management's function to determining financial allocation between corporate units and deemphasizing the need for operational knowledge) during the 1970s.
> This is especially true when the difference between either role is highly arbitrary and varies by organization or field.
I don't think this criticism is fair. Presumably if someone with the title "researcher" has a hand in actually doing what LeCun consider to be the engineer's role, LeCun would say to treat them as an engineer for the purposes of his argument.
The distinction is fundamentally pointless because the lines are blurred and highly subjective. I'll repeat it again:
>For the sake of ethical R&D, it's counter-productive to build a hierarchy of investment into the problem. Admittedly, the responsibility for end-results can differ, but the consensus that this ethical work is important should ideally be universal. That said, there should not be some ivory tower where you wash your hands of the ethical problems of your field.
Suppose you're correct, should we define this threshold based on just LeCun's heuristic for defining it? Or would it likely be better to have a consensus that your role doesn't matter in acknowledging the important of fairness and ethics?
Would you prefer the world's most prolific researchers being mindful of these issues, even subconsciously? Or to care less, perhaps very little, because it can be deferred to engineers?
Because I would like for the field to unite against building harmful systems and to acknowledge the importance of this work throughout the academic hierarchy.
The concern in your quote is separate from the question of whether we can differentiate between engineers and researches.
As for the ethical problems. Are you familiar with machine learning methods? It really is all about the data; that's not a dismissal, it's a fact about current technology. There are other kinds of A.I. which are not based entirely on raw data like this. Machine Learning has been simplistically described as "curve fitting", which I think isn't a bad description. So here you are arguing that the mathematician researching good ways to fit a smooth curve to a series of points in really high dimensions needs to somehow take into account what those points might represent in someone's use of the technology. It seems pretty unreasonable to me to require that they put ethical constraints on it.
Yes, ML research and engineering is my job. I actually lean more into the research side of our R&D function, so this isn't coming from a place of disdain for researchers.
I'm not advocating that purely theoretical work of every form should be focused on algorithmic bias. Certain realms of theory (adversarial and robust learning, deployable model theory, computer vision and NLP) lend themselves much more directly to societally-relevant bias than other realms (algorithm and complexity theory, hardware research, performance research, pure statistical learning theory). I work on faster graph neural networks, so I'm actually in latter group.
So? I just want everyone to be on the same page on the importance of work in algorithmic bias. I've seen people dismiss Gebru's work as "pure rhetoric" on Reddit - this is a cause for concern! Acknowledging the validity and importance of similar work is especially important for people who are leaders in the field and who have influence over priorities. Don't people on HN complain about FB's algorithms literally every day? Shall we forget the Myanmar incident?
Let me put it this way: in biology, Watson and Crick were researchers who participated in discovering the structure of DNA. For most of their careers, they were not practitioners. However, James Watson made a huge negative impact on the field by advocating against woman in science and advocating for eugenics (completely ignoring Rosalind Franklin). Setting an aggressive and toxic tone in genetics paid dividends during the Asilomar conference (1975), where reporters and scientists who were critical of big-name organizers got de-badged and escorted out of a conference on ethics.
Our leaders and researchers matter - let's not make the same mistakes. The message should be: "I might no longer tool in TensorFlow, but I care, and so should you." It was easy for Jeff Dean to do that, which I thought was awesome. No senseless purity testing of engineer versus scientist. I think small steps like the new NeurIPS broader impact statement are heading in the right direction.
Edit: I just realized this reads like I'm equating LeCun with Watson... that's not my intention. That would be incredibly insulting, my apologies. I just needed an example of leadership having ripple effects throughout a field. Mea culpa.
Well if you're literally researching ways to impose a bias on an estimator then sure your approach will be more susceptible to bias. Ok then, LeCun should amend his statement to say also tell the engineers not to impose a racist prior if they use some kind of engineered regularization term. Actually isn't that what fairness researchers are developing themselves? Ways to attack ML systems in order bias their behavior? Honest question.
I am personally of the opinion that it is fundamentally impossible to advance technology in a one-sided way. Anything with the power to do good can do evil too. Power itself is the danger. There might be a logical proof of this somewhere. Step very far back, and try to describe what a technology like a ML algorithm provides to society: software that can perform tasks as well as a person? discriminate between similar things using noisy observations? extract information that is obscured? The technology which accomplishes this can always be used both ways.
Do you have a citation for reporters and scientists being de-badged and escorted out of the Asilomar conference (I'm always curious about the history of molecular biology, and this is new to me).
BTW, Watson didn't ignore Franklin- her name is listed in the W&C Nature paper as providing data. What he said in his book was much worse than ignoring her.
which adds a ton of color, and also kind of supports the problem with overly enthusiastic science reporters publishing things irresponsibly early (covid reporting is a good example).
The core distinction is understanding that bias in ML is not _just_ bias in the data. It may be true that we can reduce bias in this model by reducing bias in the training data, but there is a deeper, more fundamental problem of bias that will not be solved just by changing the training data. Marginalizing the discussion by pointing out that this case would benefit from less biased data is unproductive.
> Marginalizing the discussion by pointing out that this case would benefit from less biased data is unproductive.
Except for the part where it provides an actual workable solution to the problem at hand.
To me this is an argument between completely different mindsets, one that restricts itself to provable facts and one which restricts itself to political agendas. I don't see how the latter can also work in facts. Or belongs in a technical research discussion at all frankly. You want to make laws that force companies to produce identical/equivalent outcomes for every race somehow? Just go lobby for it. Perhaps it's a good idea. You aren't going to reprogram mathematicians to think in political terms instead of mathematical terms.
I asked another commenter as well, but what are the proposed solutions then? People are obviously upset about ML and bias, is there a place I can get a summary of actionable next steps to lessen bias in ML?
There’s not one neat trick to make it go away. There have been a number of fairness and bias workshops and forums in recent ML conferences. There’s also a growing podcast and book collection on the topic. Timnit Gebru (mentioned in the OP article) has published and participated in a bunch, maybe start there.
This lack of actionable improvements or concrete guidelines reminds me a bit like needing "political officers" in Marxist military units who ensure "compliance".
> This lack of actionable improvements or concrete guidelines reminds me a bit like needing "political officers" in Marxist military units who ensure "compliance".
Sure, you can look at it that way if you like, that commonly results in hiring someone to be responsible for D/I without actually making any other changes.
A better response is more along the lines of "Not in MY Army" which makes it everyone's responsibility at every level.
There is really no way to know or to say with logical justication that a data set, or the deep learning model that results from it, is free from bias. That's the issue at hand. The social dimension of the application must be discussed broadly to make any sense of it. It's a matter of opinion, and properly so. I worship the guy and I think he was attacked for things he didn't say, yet if Mr LeCun had acknowledged the limits of dnn black box models, it would have been helpful for everyone following along. Timnit Gebru should also have made the same point.
Sort of. I didn't read the whole exchange, but to me the thing to say would be to be a bit more explicit that the tool will always reflect the bias of it's owners. Hopefully phrased to sound less Marxist, but something along those lines, instead of leaving the impression that the only issue was an inadequate data set.
Right, probably it’s better to look at the model and call it “fair model” as long as the model output error on a given class of inputs is inversely proportional to the percent of this class in training data (eg more training data of the same class means less error and less training data means more error). I can’t recall a single ML model which was not fair though I mostly deal with texts, not images.
Yes, the problem in all these discussions is that unless you engage with them on broader epistemological grounds, you’re just occluding the a priori biases.
Engineering, as a discipline, has an unfortunate history of not wanting to engage with the social and political conditions under which its work occurs, but that becomes entirely untenable for how a lot of ML is put to use (if you’re trying to be honest anyway).
Balanced how? It could be balanced to represent a population, but which population? The US? The university where it was created? The world?
It could also be balanced so that the evaluation metrics were similar for each subgroup (possibly ending up with a sample that's very different from the population). But what are the subgroups? For any commonly used definition of race, there is a lot of intra-group variety.
Maybe balancing the training data is enough. But figuring out what balance even means is a huge question.
I don't think there's a ready CS/stats solution to this problem, so it will require interdisciplinary engagement and listening to the people who have been on the wrong end of facial-recognition bias is likely a place to start.
Because ML models don’t exist in a vacuum. The intentions and biases of the people who build and use them affect which models exist, and how they’re used. Creating a perception that models are unbiased mathematical oracles because the dataset is unbiased can be used to support harmful uses.
ML models don't exist in a vacuum, but they do exist in an empirical reality. And in an empirical reality, there is always the fundamental unbiased measure of success: predicting whatever it is the model is built to predict.
And this, I think is the knife that separates the different schools of thought on the issue.
People who are judging whether an ML model is "good" or "bad" based on this criteria necessarily see the accusation of "bias" as a claim that their model is not successfully predicting things. They rightfully retort that they would do a better job with an unbiased dataset. To argue they are their models are always wrong on their terms is to argue that there is a Ken Thompson-like hack in their mathematics. [1]
On the other hand, people who judge ML models by criteria like how they might be used or interpreted by laypeople are fundamentally talking about something other than ML models-qua-mathematical models. To the modelers, you might as well be arguing that the theory of nuclear fission is biased against the Japanese. But you are not actually talking about the empirical quality of their model, and so on your own terms you are correct. The models can be used improperly, and researchers should be careful about how their findings are perceived.
Thanks for exactly this example - nobody stated yet but the charitable / best faith interpretation I can see of the Gebru angle here really is exactly that there’s a Ken Thompson hack at play, or at least a high risk of vulnerability to such a hack.
I just don’t know how one would prove it, and as others have noted I don’t understand what the mitigating alternative in the short term should be other than just stopping the research.
Fundamentally, the choice of training data set, and the biases that went into it's collection.
Also, in the case of statistical models, the crafting of the trained features themselves.
Actually, this is also relevant for neural networks despite the fact that they learn their own features because some amount of "framing" of the raw data often takes place in order to focus the neural network on the portion of the input data the trainer sees as relevant. This removes noise, but also removes context.
You asked about the biases of the people building the model, which is what I answered.
You didn't ask about the biases that occur during the requirements specification stage, or the biases that occur during operational implementation of the trained model.
Those are just as important - and arguably even more important - than the choice of the training data and the technical implementation.
The responsibility for the ethics of using ML neither begins nor ends with the ML engineer who builds the machine, and there are serious questions arising from the application of ML in certain domains that cannot simply be addressed by "better training data".
It’s not as though the designers of the system set out to train it on a biased dataset; we can assume they were trying to be balanced from the start.
And that’s the deeper problem here: “it’s just a biased dataset” is a misdiagnoses. It’s a whole system of biases that leads to people thinking they are training with balanced data when they manifestly are not.
You’re never really going to achieve this mythical “balanced training data” until you untangle all of the other implicit personal and organizational biases. There are a whole host of ethical discussions that need to happen to even begin to flesh out what “balanced” might even mean for, say, facial recognition software intended for use in law-enforcement, but the same biases that lead people to skip right past those discussions and begin training are often the very ones that result in the biased data to begin with.
What actual task are you performing that requires you to generate a high-resolution photo of a face given a low-resolution photo of a face? If it’s just for fun, then sure, bias may not matter by definition. But if it’s used to make decisions that have real consequences for people’s lives, then it sounds like a really bad idea no matter what the training set is.
Saying that the issue with the model was biased data is like saying the issue with obesity in America is that people eat too much. It's ignoring the broader context of the issue, in favour of a simple answer of "well if people ate less McDonald's, there wouldn't be an obesity crisis."
Yes, we could dive into why people eat too much, and how they could start eating less, but very soon we're going to be guessing and could easily come to the wrong conclusions.
I also think there's value in simple true statements.
If it's so easy to fix the bias, then why isn't it fixed, always? That should be the expectation.
More likely, the reason that bias isn't routinely fixed is that it isn't easy, and these kinds of biases do make it into production systems. Which makes it a net positive in my view that the occasional shitstorm reminds society of this fact.
Can that be unpleasant for AI researchers? Sure... but if it bothers then, then perhaps they could focus their research on trying to fix the problem?
Physicists had and have unpleasant conversations about their moral responsibility for nuclear weapons. Other fields of research should take their moral responsibilities seriously as well (not just AI research, by the way).
But can you control other model developers dataset choices?
ML tools are developed, and filter down into the general developer population where they are used without fully comprehending the biases they contain or can contain if used incorrectly. I'm pretty partial to the idea that ML is a systemic bias footgun. VERY hard to not use incorrectly, and with potentially HUGE social repercussions. E.g. the youtube funnel towards extremism that was well documented a few years ago - visitors start on some innocuous video, and get recommended more and more extreme things, funnelling traffic towards the alt-right in an unhealthy way.
It IS the responsibility of a data engineer / scientist to process the data, but this leads to a questions - to what lengths should one go, to ensure that the dataset is balanced?
For example - in the small town I'm from, there are practically no people of color. It's an extremely homogeneous town, in terms of ethnicity. Everyone's white - the only thing that seems to differentiate people, are their socioeconomic backgrounds. So in a way, ethnicity becomes a irrelevant feature, if we were to use it to say, profile and predict criminals.
And what's more - if we were to transfer that model, which is trained on data from my small homogeneous place, it would probably generalize very poorly in areas with more diversity.
On the flip side - if we were to make a dataset out of criminals in some large and diverse area, we'd need to get our sampling methodology right. Maybe it just so happens that the law enforcement tends to pour all their resources in policing poor areas, where some certain ethnicity is very over represented? As you can see, the further down we go, the more this problem moves from models -> datasets -> sampling -> policies, and so on.
And as for our imagined crime profiling model, you can see that the scientists are oftentimes forced to work with wildly different datasets, that may look very different from one another. You get a bunch of different models with great local optimization, but which fail to generalize on the population. And what's more, those locally optimized models may work perfectly well for their intended tasks, so there will be no complaints - until one of them are deployed to cover more general cases.
Yes - the ML engineers and scientists are responsible for building good generalized models.
Yes - the data engineers and scientists are responsible for building good datasets
But alas, there's only so much one can do about the above. In the end, the data actually stems from something real and tangible, and if there's a systemic bias in how that data is created, then that's going to be the greatest factor for everything which comes after.
It's a very deep and complex topic, which goes way beyond this. But sadly there are real-life consequences, when models are being deployed in the real world.
I haven't followed the original debate, so not sure if this was a part of the argument against LeCun, but it could be:
The issue is LeCun makes an argument about fundamental research, but is not exactly a fundamental researcher, and does not necessarily represent fundamental research.
As an analogy, if you are a researcher of general chemistry, persumably there is no issue. However, if your research is specifically about chemistry for improving bullets, and you produce working prototypes, then some might say you should be subject to some regulation as a part of the arms industry. I'm not saying this is the right thing, just that such a point could be made. LeCun is arguably much more the second kind of researcher than the first. The research he represents is "better ways to recognize faces", not "statistical properties of natural images".
To take another example, there is an enormous amount of regulation in, say, medical research. And in this case there are good reasons for that. Gebru could be possibly arguing for something similar in say face recognition.
My takeaway is that people disagree that the problem is simply the choice of dataset, but rather, they think that the problem is using ML at all to perform certain tasks (including, perhaps, making important decisions based on a person’s appearance).
Great link, the one part of LeCun's argument I don't understand (or maybe I disagree with it) is that in Tweet 9 he minimizes the effect of architectural bias mentioned in Tweet 6 by saying that modern DL models have "generic architectures". But his own Tweet (#11) apparently demonstrates a problem with that reasoning; modern DL architectures are bad at generalizing from rare categories, which would seem like an architectural issue that causes significant bias, especially in the sense of racial bias. I think other people may have touched on something similar by pointing that bias might be inherent in the metrics researchers optimize for when designing new models.
I agree that some people appear to believe this. But what is the justification for it? I don't really get how it cashes out in concrete harm, assuming you de-bias the datasets.
To be clear, I do get arguments that ML might increase the efficiency of policing, and there is some inherent tension between the efficiency of the state and freedom, to the extent that imperfect enforcement is a tacit feature of our legal system. But, this is not an inherently racial point.
But how do you de-bias that datasets when the world itself is biased? You can't just ignore that bias in the real world because any ML will at best mirror that bias if not amplify it.
For example imagine some ML algorithm that predicts the potential value of a customer (or their likelihood to steal from you) as they walk in the door of your business. How do you remove the racial bias from a system like that when racial biases exist in society that result in the average White person being in a higher socioeconomic class than the average Black person? The ML algorithm will tell you White customers are more valuable than Black customers. It is doing exactly what you told it to do, but the end result is that Black people get worse service in your business.
You're talking about a different problem than the original thread. The original thread isn't saying "the underlying world is biased"; Obama's face exists in the world.
It's saying that the dataset isn't reflective of the world, and this is an issue because then you'll end up with voice tools[1], facial recognition software[2], and sentencing algorithms[3] end up performing disproportionately poorly on some minority groups. And this ends up reinforcing the poor positions many people are already in.
The way I understood his original tweet was that he was taking the specific problem with this study to illustrate a point about ML overall. Therefore the criticism he receives can be related to how that idea applies to ML overall and not exclusively this problem.
Phrenology was bad because it was scientifically invalid. I don't see any problem with predicting traits from appearances, at least, not intrinsically if the data is accurate.
> Phrenology was bad because it was scientifically invalid.
Well, that's not a very full story. It was scientifically invalid because it modeled mental traits based on the shape of the skull, and that's the important part. If you trained a model to predict traits based on accurate data about skull shape and traits, I'm confident that the algorithm would happily do that. Phrenology would still be bad regardless of what technical methods are used to do phrenology.
What is your argument that it's still bad? That was kind of a non-sequitur. If we have a model that takes in skull measurements and accurately predicts whether you are an introvert or extrovert...what is the problem with that?
There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing.
I really don't want my field to regress towards 19th-century pseudoscience when we can empower other scientific advancements that will yield immediate benefits.
> There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing.
This is a strange point. Are personality types subjective? Is IQ? Maybe they are, but we still sometimes find them useful to quantify. If it's ok to predict these things from your answers to questions...why isn't it ok to predict them from pictures of your face?
Ignoring the (many) arguments that IQ is not really a great marker for intelligence, there's basically no value in trying to predict IQ based on facial structure. Unless it is quite literally 100% accurate (which it is not), you will end up mispredicting people's "intelligence". Even if this is ultimately just randomly distributed (and it probably won't be), you'll be needlessly treating people unfairly for no reason.
So yes, I think I can state pretty concretely that there's no value in trying to predict IQ from face shape.
> Ignoring the (many) arguments that IQ is not really a great marker for intelligence, there's basically no value in trying to predict IQ based on facial structure. Unless it is quite literally 100% accurate (which it is not), you will end up mispredicting people's "intelligence". Even if this is ultimately just randomly distributed (and it probably won't be), you'll be needlessly treating people unfairly for no reason.
The presence of error in a prediction makes that prediction valueless?
When doing anything prediction-y, you need to take the relative value of false positives and negatives into account.
Can you describe a situation where the harm caused to someone by mis-classifying their IQ is outweighed by the increased efficiency from...being able to more quickly identify someone's IQ? Like, can you even describe a situation where this will be used ethically at all? What use is there for predicting someone's IQ with dubious accuracy, assuming you already have their permission?
Sure. But you also need to account for the actual baseline you're comparing to. Whatever process it might be that's taking someone's IQ as an input is right now being handled in some subjective manner by other humans. Do you think their error rate is zero? These things don't need to be perfect to be good. They just need to be better than what we're doing now. And as per all the recent unrest, it turns out that what we're doing now isn't all that great.
> Whatever process it might be that's taking someone's IQ as an input is right now being handled in some subjective manner by other humans.
Yes.
> Do you think their error rate is zero?
Well it depends. Do I think the error rate of IQ as a predictor of underlying intelligence is flawed? Yes.
Do I think that IQ as a measure of IQ is flawed? Still, often, yes.
> These things don't need to be perfect to be good.
To my point above, I generally am not a fan of IQ in general. Psychology researchers are consistently finding new and interesting ways that IQ tests are socially/culturally/environmentally influenced and that exams may not be fair.
If your claim is that you can somehow train an algorithm to look at someone's face and evaluate their underlying intelligence better than the tools we have today, honestly "delusional" is the first word that comes to mind. That's just not something that currently available tools can do. And thought experiments about a "perfect" classifier aren't interesting when we're discussing applied ethics.
And that comes with the huge, enormous, gigantic, asterisk that you can even measure "intelligence". Like, it comes with the caveat that you can even reasonably define intelligence. We generally measure intelligence as correlated with success at some metric: math problems, visual puzzles, chess, reading comprehension, whatever. If you're contending that a hypothetical model could predict underlying intelligence better than the measures we have today, how would we even know? When training a model you have an evaluation function.
If the evaluation function is flawed (which is essentially your contention, and which I absolutely agree with), the trained model will exhibit biases that reflect the flaws of the evaluation function. An ML model isn't suddenly going to solve the cultural issues we have with measuring IQ. It will encode the same biases that society does, because the model will try as hard as it can to do exactly what the human proctors would have done.
These are exactly the kinds of broader ethical questions that Timnit points out we need to worry about.
IQ isn't the point though. I'm not a huge fan of IQ specifically either. Your original objection was intrinsic to the idea of detecting things from faces. These objections are to the quality of the measures. I'm not going to defend IQ as a useful measure, because it has a ton of problems and that's not my point here.
> If the evaluation function is flawed (which is essentially your contention, and which I absolutely agree with), the trained model will exhibit biases that reflect the flaws of the evaluation function. An ML model isn't suddenly going to solve the cultural issues we have with measuring IQ. It will encode the same biases that society does, because the model will try as hard as it can to do exactly what the human proctors would have done.
Of course. But so what? There are people using IQ now for things. An ML model isn't going to magically make those biases worse, either. What it's going to do is bring them to the surface, so that they are quantifiable and we can actually do something about them.
ML is the solution to the bias problem. Right now these evaluations are being made by other humans. Humans who we cannot statistically debias. Humans who's biases we can't even effectively interrogate. The reason people are making all these memes about biased AI is not that it's more biased than humans, it's that the bias is more measurable.
> An ML model isn't going to magically make those biases worse, either.
Well, actually, research shows that unless great care is taken it absolutely can. If you include for example race as a factor in a model, it can learn non-causal correlations between race and whatever the objective is. This can have a compounding effect in some cases. [0]
> What it's going to do is bring them to the surface, so that they are quantifiable and we can actually do something about them.
I don't follow this. If we have a biased objective function, the model won't surface any biases we weren't already cognizant of in the objective function. And they were already quantifiable: we had a function that we were using to evaluate the model. We could use that same function on whatever non-model evaluation we were doing.
> ML is the solution to the bias problem.
This is basically directly in contradiction to what leading experts on the subject say. ML cannot fix bias in human systems, unless we presuppose that those systems are biased, in which case we can often address the bias in the human systems directly without ML.
> Humans who we cannot statistically debias. Humans who's biases we can't even effectively interrogate.
You can still have decisions be made by objective expert systems without complex ML. If you want to learn someone's IQ, the best way is to debias the IQ test, not to try and infer it from their face bones.
> it's that the bias is more measurable
If we can measure the bias in the output of an ML model, we can equivalently measure the bias in the output of a human system. You're presupposing the existence of some unbiased objective function which we don't have, and that's at the core of the issue.
[0]: https://www.wired.com/story/ideas-joi-ito-insurance-algorith... has a few good examples here, like how naive bail and sentencing models encode racial bias that isn't present in humans. And to be clear the response here shouldn't be "well let's just build better models" but "why do we think a model will improve the situation here at all"? Removing agency from Judges has historically been bad for the average person convicted of a crime. This doesn't mean that individual judges can't make terrible rulings, but that the alternatives are usually worse on the whole.
> I don't follow this. If we have a biased objective function, the model won't surface any biases we weren't already cognizant of in the objective function. And they were already quantifiable: we had a function that we were using to evaluate the model. We could use that same function on whatever non-model evaluation we were doing.
We can actually follow the logic of the model. For instance, you can theoretically de-bias a dataset by building a racial classifier from it. What you need is an objective test for the presence of racial information, and that's easy to obtain: Build a classifier to explicitly predict race from your feature set. Train an adversarial model to reconstruct your dataset with maximum fidelity, subject to the constraint that race can no longer be predicted from it.
> This is basically directly in contradiction to what leading experts on the subject say. ML cannot fix bias in human systems, unless we presuppose that those systems are biased, in which case we can often address the bias in the human systems directly without ML.
These experts are just wrong, then. Naive ML won't fix bias in human systems, but that doesn't mean we can't use ML to fix it, if we do so thoughtfully.
> You can still have decisions be made by objective expert systems without complex ML. If you want to learn someone's IQ, the best way is to debias the IQ test, not to try and infer it from their face bones.
Sure, but there are a lot of things that we don't do in the best possible way because it's too expensive. There are lots of use cases for cheap, scalable, low precision models.
> If we can measure the bias in the output of an ML model, we can equivalently measure the bias in the output of a human system. You're presupposing the existence of some unbiased objective function which we don't have, and that's at the core of the issue.
Right, but we cannot fix the bias in a human. And humans are heterogenous and inconsistent. The same person may be more or less biased on different days. The ML model is consistent, and we can incrementally improve its bias in tangible and testable ways. The same is not true of humans.
And what does this get you? Let's look at a face recognition dataset. What happens when you debias it? Is it still useful? No. Because the faces no longer resemble real faces.
> These experts are just wrong, then
Perhaps, but you aren't making a strong case for that.
> There are lots of use cases for cheap, scalable, low precision models.
That involve facial recognition?
> Right, but we cannot fix the bias in a human
We don't need to. We just need to fix the bias in the system. And we absolutely can incrementally reduce bias in systems that involve humans.
> And what does this get you? Let's look at a face recognition dataset. What happens when you debias it? Is it still useful? No. Because the faces no longer resemble real faces.
Not to you. But you can remove the racial information without destroying all the information that a model can detect.
But when racial information is correlated with the output, to decorate with race, you destroy the input. This is most obvious with a face dataset, but is true with anything race correlated: credit scores, where you live, etc. If you're willing to destroy the training data so it no longer resembles real world information, you might as well just not use it in the first place.
That's what the ethicists say: don't use facial recognition models. Don't work on them. Don't research them. They cannot be both unbiased and useful. And in general, there's few to no uses that are ethical, period.
Well, the ethicists just don't understand the models, then. For instance, there are a bunch of measurements you can take of faces to identify people, if you were doing it manually. Things like pupillary distance, canthal tilt, nose width, etc.
Some of these correlate with race. But only part of the information correlates with race, not all of it. It is, in principle, possible to remove the information that identifies race without destroying the information that identifies the individual. It is true that part of an individual's essential characteristics are their racial characteristics, but it is not true that the only way to identify an individual is their racial characteristics. For instance, there is no way that i'm aware of to infer race from fingerprints, but you can absolutely identify a person by their fingerprints. So, the question is, can we extract a facial fingerprint that identifies a person, but not their race? I think the answer is almost certainly yes, and it is going to be up to a clever model design to do it. But essentially it would look like a GAN where the adversarial component is constantly trying to predict race, while the Generative component is trying to trick the race classifier without tricking the person-identifier.
> Well, the ethicists just don't understand the models, then. For instance, there are a bunch of measurements you can take of faces to identify people, if you were doing it manually. Things like pupillary distance, canthal tilt, nose width, etc.
Or perhaps they understand that this won't work in practice.
I know. My point is not about IQ. My point is that people find IQ to be a useful measure of things, in certain situations. We can use something else, like the big 5 personality inventory, if you like. The point is just that subjective characteristics of a person are things that we quantify and sometimes care about measuring.
And the times when we find ourselves wanting to quantify and measure them in order to make decisions that impact people’s lives them are precisely when we need to be extremely careful, because they are subjective traits.
Of course we should be careful. But we should also keep in mind what the baseline is. The baseline is subjective evaluations by other humans. If we don't build models, it's not like we're going to live in a world where nobody's subjective personality traits are evaluated. They still will be. They'll just be evaluated by other humans in an opaque, un-monitorable way. Using models at least makes it transparent, and something that we can iterate on to remove bias. There are no effective ways i'm aware of to de-bias humans.
You have it exactly backwards though. Its humans that obstruct accountability, because they are opaque and inconsistent. ML is consistent and inspectable. It cannot lie to you about its motivations. It is infinitely more accountable than any human system ever will be.
There are lots of things about appearance that are deliberate signaling. It's all subjective but it's not meaningless or pseudoscience to read those signals.
Well, what are you going to use this technique for? What’s the use case for generating a high-res photo of a face from a low-res photo of a face? It’s hard to think of any use case with real-world consequences that wouldn’t be a bad idea regardless of the training set.
> We have noticed a lot of concern that PULSE will be used to identify individuals whose faces have been blurred out. We want to emphasize that this is impossible - PULSE makes imaginary faces of people who do not exist, which should not be confused for real people. It will not help identify or reconstruct the original image.
This is just another way to generate realistic faces that match a pattern. The "bad use cases" I can imagine involve unmasking real people, which is not what the model does.
My org allows profpics for email or messaging or stuff like that. Some people have old, tiny profpics that are pixellated at various sizes. This technique could help the pictures look nicer.
How do you de-bias datasets, when the raw data (coming from the real world) starts biased to begin with?
For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police.
So ML takes those biases, and entrenches them further. Worse yet, because people tend to believe that "computers cannot be biased", the fact that it's ML that's producing a specific result, is taken as prima facie evidence that said result is immune to criticism from that perspective. And with most advanced models, you can't really crack it open and make sense of what's inside - so you can't easily prove that it is biased.
> How do you de-bias datasets, when the raw data (coming from the real world) starts biased to begin with?
I can think of at least one trick that might work. Build a classifier capable of identifying race from the data, and then reverse its gradient to remove its signals. This would deracialize a dataset, so that there is no detectable latent information in it.
However, whether or not my particular solution works is somewhat beside the point. It's an engineering challenge to figure out how to do this, but that isn't an argument that ML is fundamentally bad for these purposes.
> For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police.
I totally agree that this is a problem. But it seems to me like the solution is more effort on fixing the biases, not reverting to human judgment. I think it's always important to make sure we understand what the baseline is we're comparing to. Humans are racist. Humans created the biases in these datasets. These algorithmic tools just bring those biases to the surface and make them quantifiable. It seems to me that that represents an incredible opportunity to actually fix the biases in a measurable way.
I’d phrase it that “bias” in the social sense can occur in (1) the task itself based on how it’s defined, (2) preexisting within the problem domain, so representative datasets could still be biased, and (3) in when the system and the people operating it choose to use the outputs of the model. Which is all to say that “bias” problems related to the model can extend beyond the model itself.
The solution isn’t necessarily to fix the data or the model. It might be banning use of facial recognition by police, for example, which some places are doing.
This was the kind of reasoning that led gun makers to develop safety mechanisms.
It isn't clear what a similar safety mechanism would look like for an ML algorithm. Modern ML algorithms are nothing more than generic pattern extractors. What would a safety mechanism look like?
And led to government regulation of distributing guns.
Have government grants for ML research be contingent on training on diverse datasets. Fund research into more diverse benchmarks and push them as the goal to beat.
It is unethical (and, in many places, a crime) to sell a firearm to somebody whom you know, or reasonably suspect, intends to use it for criminal purposes.
Yann was "technically correct" in that the nature of the results are highly driven by the nature of the dataset you use.
However, that framing of the discussion can easily be interpreted as absolving the ML community from the ethical discussions around the nature of the datasets.
Is it acceptable for the community to hand-wave these issues because they are "dataset problems"?
For what it's worth, this is a much larger debate that is happening in many fields. For example, a bunch of decisions around car safety were based on crash tests. Those crash tests are based on dummies that were designed around male body forms and thus don't really test how women would handle the crashes. To quote the author of Invisible Women:
> As a result, if they are involved in a car crash, women are more likely to be injured – 47 per cent more likely to be seriously injured and 17 per cent more likely to die.
This general questioning about the implications of bias in the datasets is happening in many fields
I think one issue here is that Twitter makes it hard to have good conversations because it motivates people to optimize for pithiness and likeability/retweetability over substance.
That said, I'm disappointed at the lack of professionalism shown by Timnit Gebru in this conversation. I understand that she's frustrated that mainstream AI/ML research doesn't have a good understanding of how models can reflect and reinforce systemic racism. But her response to Yann doesn't attempt to find common ground, doesn't add any meaningful facts to the conversation, and is ad-hominem ("why are YOU too dumb to understand this?"). Timnit is a Research Scientist at Google who leads a group that works on Ethical AI. In that role I'd expect that her mission would be to elevate the collective understanding of ethical problems in AI, and solutions to those problems. Her communication here doesn't achieve that.
This is important to me because I'm also a (fledgling) member of the AI/ML community, and I want us to be able to have civil and constructive conversations.
I think it’s also the way people use it. Are you interested in truth or power? It seems Twitter attracts a lot of people who only feel powerful on twitter, and they prioritise that over truth.
> I think one issue here is that Twitter makes it hard to have good conversations because it motivates people to optimize for pithiness and likeability/retweetability over substance.
Which is why it's probably a good thing on balance that Yann LeCun left Twitter. Blog posts and comments are the bare minimum for an adequate forum for discussions like this, since the reader is at least encouraged to read and understand an accurate and complete summary of the author's points before making a pithy reply[1].
[1] My phrasing does not necessarily refer to any of the figures involved in this conversation.
Fixing something in AI usually means improving it, not solving it. And even if the problem would get solved, there are plenty of other problems to work on.
Maybe not for her. Seems her education is actually electrical engineering for some reason, and her Wikipedia page lists no record of AI research achievement beyond racial/sociological stuff. If she accepted that the USA/AI isn't filled with racism she'd presumably have to go back to signal processing.
FYI: education in EE is actually pretty common for people in ML. The math needed for control and signal processing has a lot of overlap with ML, and many EE labs do research in ML. I don't agree with her responses to LeCun, but I don't think her education should impact her credibility.
Edit: If you read her bio, she received her PhD in CS, and it looks like she's published multiple papers on things other than social issues in ML. What are you even talking about?
Citation needed. I don't like this sensationalism.
"civil and constructive conversations."
What was uncivil nor constructive? One person blamed a dataset for bias, but said researchers should not be on the hook for consequences stemming from the output that could be put into a deployed product. The other disagreed with the notion and stating that this was brought up in the past showing clear frustration. Uncivil tends to be a dogwhistle indicating how a person SHOULD act as if there are strict set of parameters for how to show frustration.
> Yann, I suggest you watch me and Emily’s tutorial or a number of scholars who are experts in this.
In other words, poor Yann should be sent for reeducation!
> Maybe your colleagues will try to educate you. Maybe not. But I have better things to do than this.
Or maybe not. So much thinly veiled despise. If it weren't for Yann there would be no computer vision as it is today, and her job would be the realm of sci-fi.
> If there is any sort of fairness, ethics or what have you initiative @Facebook and @facebookai, let me tell you how little credibility they have because you have the loudest microphone out there.
Now switching on an indirect strategy - throw shit at his employer in order to force his hand.
> I’m sick of this framing. Tired of it. Many people have tried to explain, many scholars. Listen to us. You can’t just reduce harms caused by ML to dataset bias.
Yet I have hardly seen a good counterexample of algorithmic bias, it was almost always the data. Gradient descent doesn't know about race. CNNs, RNNs, Transformers - they are all orthogonal to social bias. Maybe regularisation can have an effect on accuracy for small (less represented) classes, but not much more.
"""
Known for her work on racial and gender bias in facial recognition systems and other AI algorithms, Gebru has been advocating for fairness and ethics in AI for years. The Gender Shades project that she leads with MIT Media Lab Computer Scientist Joy Buolamwini revealed that commercial facial recognition software was more likely to misclassified and was less accurate with darker-skinned females compared to lighter-skinned men.
Gebru’s CVPR 2020 talk Computer vision in practice: who is benefiting and who is being harmed? again addressed the role of bias in AI, “I think that now a lot of people have understood that we need to have more diverse datasets, but unfortunately I felt like that’s kind of where the understanding has stopped. It’s like ‘let’s diversify our datasets. And that’s kind of ethics and fairness, right?’ But you can’t ignore social and structural problems.“
"""
Seems that these two paragraphs alone describe what she has been saying. Where did your "algorithmic bias" assertion came from? Who mentioned algorithmic problems?
> If there is any sort of fairness, ethics or what have you initiative @Facebook and @facebookai, let me tell you how little credibility they have because you have the loudest microphone out there.
Its interesting how one perceives this as "throwing shit" when it is harsh criticism directed at what she denotes as the little power any ethics or fairness org at FB has.
The other quotes are out of frustration and are in the civil territory. If those are uncivil to you and the others, well, that's your line of sensitivity as it is definitely not the bar for incivility. If something sounds harsh, it doesn't mean it is uncivil. It is very easy to twist words or remove them out of the conversation context to make them seem like fighting words.
I could be very harsh in address people here and still be civil about it. The air I smell here 'It is Yann LeCunn, legendary figure, never criticize what he says or if you do so, do it very very very politely even if you have to do it 1k times.'
I'm no longer interested in finding out more perceptions on this. I saw what I saw and you guys see what you see. I don't see any uncivil activity just sensationalism at an opportune time and a mob of anti-social justice trying to attack whatever seems "woke." These issues would have to be dealt with no matter what period of time.
You are correct about the 'quote', as it does not seem Gebru ever wrote that. I am surprised none of these sibling comments have affirmed that. While personally I presume that parent poster meant the 'quote' only as an illustrative pharaphrase, it is clearly very easy to misinterpret. (Of course, if one assigns less charitable intent it is intentionally misleading, but I generally assume the best of people.) Either way, it feels intellectually dishonest that no one else has conceded this.
As an aside: I wish English had some more elegant method for forming compound words than just-concatenate-words-with-hyphens. Compounding of phrases and use of imagined quotes as compound words is one of those common flourishes of verbal communication that is difficult to express in written English.
As a note: I believe it is primarily the "dogwhistle" reference is what is earning you down-votes, as it subtly implies — whether intended or not — that parent poster is somehow racist. I presume the best intentions and up-voted you, but keep in mind how it can be interpreted.
"why are YOU too dumb to understand this?" is not a quote and was not meant to be; it's a paraphrase/interpretation of what she wrote. I think this is fairly obvious in the context of my comment.
I'm glad there are a few people who looked at what I said objectively. I only asked for citations and noted my reason for asking. Had not seen one citation but sensationalism and perceived attitude from text.
I've had people call my behavior "unprofessional" in the past because they had a very narrow set of expectations for "professional." Whatever that seemed to fit them per person. Always glad to have a third party to sort it out with them. I regard unprofessional as a potential dogwhistle these days since the only people who understand what it really means are the ones who knows it is being used in a narrow subjective manner. They are just loaded terms.
People who cry dogwhistle instead of addressing the point tend to be more interested in shaming people for accidentally saying something undesirables say than engaging in constructive discourse.
Who cried dogwhistle? You can perceive what you want, but that does not make it true. I'm not going to go back and forward on this kind of pedantry. I asked a question, requested citations for alleged perceptions of incivility and will only respond to answers to it.
You. There is no point in telling people that there are dogwhistles in their argument. The extra woke points it gives you don’t help with people interested in logical discussions.
> You can perceive what you want, but that does not make it true.
I’m not perceiving anything. It’s a term you used right in your comment. I’m just telling you that it’s a term used to divide and it does nothing to support a broader point.
The critic here, Timnit Gebru, is not doing a good job of articulating a well-reasoned argument about why dataset bias is not the only problem with ML. Allow me to attempt to do her job:
Machine learning approaches are powerful because they can take a “small” set of labeled data and use that to make billions of new inferences at low cost. The problem is that even when these data sets are fully representative of the “real world,” the “real world” is rife with biases and unfairness and machine learning takes these unfairnesses and gives them a huge Tensorflow-powered megaphone.
For example, let’s say you wanted to use machine learning to create a system for employers to make sure that they aren’t hiring likely drug dealers. If you get a data set of every single American drug dealer’s mug shot, you’d get a disproportionately black population because white kids selling ecstasy to their high school classmates rarely get arrested and booked. The data set is not “biased” —- it is complete —- but the enforcement of the law is biased. But now, rather than using the development of this software as a moment to remove unfair bias, a machine learning model will amplify it, causing lots of black men to be denied legitimate employment opportunities, which will actually increase the chance that they sell illegal drugs to support themselves.
This machine learning doesn’t just amplify the biases in the data set, it amplifies all of the biases and other imperfections in all of the events that fed into the creation of that data set.
I feel like this is really unhelpful nitpicking on the definition of dataset bias. In your example the dataset you want is the one of drug dealers so the dataset from the convictions IS biased and uncomplete.
Well this is a contrived example. But the fact is that all datasets contain the full biases of the world that created them. So there is no "unbiased" dataset... meaning that powerful AI amplifies the world as it is, rather than a designed point of view of how the world should be. That's fine in some cases (e.g. tumor detection in X Rays -- the combined judgments of thousands of surgeons might truly reflect how we want to treat a patient) but may not be better in other cases (the combined judgments of thousands of police officers may NOT be how we want to treat a suspect).
And imo LeCun would agree with this, they were talking past each other and I don't get the behavior of people lashing out and forcibly getting everyone to bend to their opinions.
>it amplifies all of the biases and other imperfections in all of the events that fed into the creation of that data set.
A common retort is "well then, fix the data set, or correct for the bias."
But a further problem, to riff off what you said, is it's not just this one data set, it's all data sets. And while a researcher or engineer may be able to make choices about their own data sets, they won't be able to control all data sets.
This problem is also embedded amongst other problems such as the tendency for people to "trust the computer," which sounds naive, but is sometimes restated, amongst smug people who think they aren't naive, as "trust the algorithm."
If this hypothetical system's goal is to prevent convicted drug dealers from getting hired, this seems like a perfectly complete and unbiased data set.
I guess Yann is making a statement about Twitter being a toxic cesspool or whatever. The thing is, in other public forums where he will make himself available for open conversation, he will experience the same thing. When you are Chief AI Scientist at Facebook, you are the face of Facebook, whether you like it or not.
The criticisms leveled at LeCun has nothing to do with Facebook or his employment, but rather with his commentary.
To summarize (in case you didn't read the article):
> LeCun responded, “ML systems are biased when data is biased. This face upsampling system makes everyone look white because the network was pretrained on FlickFaceHQ, which mainly contains white people pics.
> “The consequences of bias are considerably more dire in a deployed product than in an academic paper,” continued LeCun in a lengthy thread of tweets suggesting it’s not ML researchers that need to be more careful selecting data but engineers.
Other AI scientists, most notably Google's Timnit Gebru, disagreed with the framing of the situation strictly in terms of dataset bias:
> Research scientist, co-founder of the “Black in AI” group, and technical co-lead of the Ethical Artificial Intelligence Team at Google Timnit Gebru tweeted in response, “I’m sick of this framing. Tired of it. Many people have tried to explain, many scholars. Listen to us. You can’t just reduce harms caused by ML to dataset bias.”
It's an especially sensitive time now when AI is being used to power many judgments with real-life consequences.
On twitter people are trying to impress other people through one upping or being the wokest more so than in person.
So, in person, people may stand up and shout, but moderates in the crowd will give them a minute to say their peace and then drown them out, if they drone on...
Interest point. So whilst COVID-19 leaves us all stuck on twitter... we can expect the quality of debate to get worse until in-person discussions resume and act as a corrective.
The level of discourse on Twitter is noticeably worse than every other major social platform (except YouTube comments if you count those). I'm sure people will have tough questions for him elsewhere, but the sheer nastiness is a distinctly Twitter phenomenon.
Sure, but he won’t be able to comment on shit and ignite another controversy, like what seems to happen on Twitter.
People like him usually communicate to the world through conferences, papers and statements where the messaging is reviewed. Twitter let’s people speak their minds, and you know what? Most people who have not dealt with issues of racism and bias just don’t understand how to have a conversation about these things and screw up.
I've been following Jeff Dean's twitter for about a year. Even if we put aside the technical contributions that he and his teams have made, that man is a paragon of leadership. Compassionate and yet so level-headed and reasonable.
Yes I agree with you but what’s the point? Everyone should be like Jeff Dean? Many people don’t have those qualities and don’t desire to have them.
I have no idea who this person is, but it sounds like he’s not very good at engaging with the public on Twitter, so the best strategy for him is to leave the platform.
I can't help but shake my head at this whole story.
Someone (who by the way describes themselves more as an artist than anything else in public bios) builds a ML "bullshit artist" (literally) that hallucinates portraits from ridiculously undersampled data. Someone else runs a low resolution image of Obama through a hallucinating computer program, and gets a hallucination.
The story should end here at "these hallucinated portraits have basically nothing to do with the input data, they're meaningless". I would bet that input images that are this low resolution are difficult or impossible to determine the race of, no matter how diverse the training data for this model could have been. Note that the skin color of the low resolution Obama and the random "white" guy it generated are basically the same!? I doubt this pipeline will ever be capable of not confusing the race of its input and output.
What is on display here is the uselessness of this ML pipeline, not its racist nature. I have seen probably a dozen of examples of this ML pipeline failing hilariously in ways that have nothing to do with race. Yet this failure is suddenly fascinating to the entire tech journalism space as an example of ML algorithms being racist?!
And somehow, despite the absurdity of this situation, even ML "luminaries" get drawn into a destructive whirlpool of social justice debate about a hallucinating computer program that doesn't even work sometimes? What the fuck are we all doing here?
If you look at this single instance, independent of the world around it, it looks ridiculous.
You need to look at it in the context of a long[1] list[2] of works[3] detailing this exact problem, and the frustration that an leading expert in the field still doesn't take it seriously, despite the real-world consequences of that attitude.
Why not pick all those other hills to die on, then?
If you want to talk about statistical methods being a horrible idea for policing, I'm with you. Probably most machine learning experts would be, too, just like most software engineers don't trust electronic voting. Instead, people chose "dataset bias affects machine learning models", the most basic Machine Learning 101 fact ever, to try and debate against.
It's like the woke people seek out the weakest possible ground because it generates in-group opposition that they can fight with and out-woke. Stronger ground could have national consensus, way beyond bay area consensus.
Actually, that's Gebru's point. Gebru was pointing out frustration that LeCun was focusing on this narrow question of statistical bias, rather than the broader social issues you're getting at.
Your comment, among the sea of others, got through and made some sense to me. My follow-up questions are:
Does that negate the truthfulness of LeCun's response?
What would Gebru rather he do or say?
I feel like I can't tell what the desired form of action is from the article, and I'm unsure what good it does for removing bias from ML to attack prominent figures (on twitter let alone anywhere public).
> Instead, people chose "dataset bias affects machine learning models", the most basic Machine Learning 101 fact ever, to try and debate against.
No one disagrees with that statement. The point that the ethicists are making is that that isn't the whole story, and that focusing only on the lowest hanging fruit (dataset bias) is ultimately harmful since it shifts focus away from the other, deeper problems.
I'm not clear about how Yann does not take it seriously? He even admitted respect for Gebru's work. He's just making a technical claim that this is due to data bias. Whether that's true, who knows, but I really don't see how he is being callous in all of this.
I'm not sure I understand what the issue is at all.
"Garbage In, Garbage Out" apparently dates back to 1957. I remember the expression from the very first computer book I read in the 80s.
The idea that this concept, one almost as old as computers and practically a theorem of computation, would somehow be racist or trying to paper over racism is ridiculous.
Narrowly scoped to GIGO, the concept can be traced more generally back to Charles Babbage in the 1800s.
> On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
I couldn’t tell you when the discussion about bias and unreliability in ML training sets started, not my field of expertise.
GIGO, which in this context would translate to something like "using a bad dataset means there is a problem with the data, not that the tool is racist". I'm not an expert, but I think the argument here is it's far far too easy to have a "garbage" dataset with these tools. I.e. one that only functions correctly for some racial groups (like this one does). That's only racist if you then try to use it in a situation where it comes in contact with multiple racial groups - like, virtually anywhere in the real world.
I think you're right that this algorithm doesn't do a good job of reconstructing high res faces from pixelated ones, simply because it can't—there simply isn't enough information in the input, and there is a huge set of real people (from many races) that could map to a given pixelated image.
But I do think that when someone who is white (or another race) says "here's a cool thing, check it out" and someone who is black (or another race, different from the first) tries it and realizes it doesn't work for them because of their race, there is a bit of inadvertent racism there. There is an inadvertent exclusion of that second person. I think this happens a lot and in some cases is unavoidable (e.g. language is intertwined with race, and there is a vast amount of information—literature, news, art—that will be inaccessible to you depending on which language(s) you speak), but it is a problem, and it deserves fixing when possible.
The implications in this case are currently small—it's a demo. But there is a risk that these biases could make it into real systems and have real impact on people's lives, and to combat that we need more awareness of this effect in the ML community and more research into possible solutions.
Whatever. I'm sure LeCun will be dying for another job.
Gebru undermines her own cause by being a shitty ambassador. While asking for deep empathy in a radically marginal area of inter-sectional study, she seems to lack any of her own. I'm sure berating an adult for a technical assertion - one that is prima facie true - is an excellent tactic that has worked to change minds and open up insightful public discourse.
Gebru's commentary smacks of narcissism. Want a little air time and some notoriety? Call a famous person a bigot and see what happens. She has little to risk and much to gain.
Agreed, it bothers me that this tactic is so successful. I could not find an instance in this "discussion" where Gebru outlined any concrete steps to solve for, or lessen bias, in ML. It seems she mainly wants to complain, rather than fix the problem. I have been googling about ideas to lessen ML bias, and the best I've found is "diverse training datasets" which Gebru herself says "is not enough".
Well what is then? Outrage without a solution is useless.
The discussion mentioned the Tutorial on Fairness Accountability Transparency and Ethics in Computer Vision at CVPR 2020. The tutorial was recorded in three parts. The third part[1] includes solutions. These are either direct quotes or paraphrases of the bullet points contained therein.
2. Recognize [l]imitations(sic) of technical approaches (this discussion)
3. Model documentation frameworks[2]
4. Standardized framework for transparent dataset documentation [6]
5. Positionality awareness [7]
6. Actively follow the perspectives of people in marginalized groups
7. Make intentional design choices to privilege the perspectives of marginalized stakeholders who are at most risk of being harmed by the technology we develop
8. Value interdisciplinary and 'non-technical' work
Note: These are some of the sources given as examples. Some are omitted for the sake of time.
Also note: Just because there are points doesn't obligate you to agree with them
To clarify, The Tutorial on Fairness Accountability Transparency and Ethics in Computer Vision is not a technical presentation, it is very much a intersectionalistic take on the topic, more concerned with power, oppression, and social justice than engineering.
Suggested technical solutions [1 above] was exactly one sentence and the only sentence in the presentation that suggested any sort of technical approach, presented without elaboration, and the recommended solutions like Counterfactual Testing are also methods for fixing the training data.
The remaining 99% of the presentation was spent on the idea that if AI engineers and scientists were more diverse and attended more diversity training, the models they produce hopefully wouldn't be biased.
Suppose I dismissed cryptanalysis by suggesting anyone involved in breaking cryptography ought to produce better themselves.
Suppose I dismissed a piece of medical research that shows some drug is no better than a placebo, and I demanded that those researchers produce their own drug that works better than the one they presume to criticise?
Critique is valuable. The entire field of science itself is built on absorbing critique, and checks and balances to ensure we reason out of robustness rather than hasty assertions.
In other words: suppose Gebru identifies that some particular ML model is racially biased. That insight, if true, is in itself a valuable contribution to the field. Suppose Gebru further develops an argument that the nature of this bias is (or is not) one in which different choices in training data will not substantially solve. That, too, is itself a valuable contribution to the field, and this kind of work is not at all the same thing as pointless "outrage".
I think the Yann's main point is that this specific model's racial bias was a training data problem; Gebru has not provided any evidence otherwise, as far as I know, apart from repeatedly asserting that better training data won't fix the problem.
It would indeed be a valuable contribution if Gebru could have shown how better training data would not have fixed the problem (e.g. by feeding better training data to the same model and still reproducing the issue).
But as it is, she is asserting that better training data won't help without proof and not recommending any alternatives, while taking a hostile tone of conversation. I would be hard pressed to take her claims as good faith criticism.
Adding some more sources to show that these scholars are not raising issues without concrete solutions -- Gebru and others have proposed many solutions and more concrete problems than this easily solved question of statistical bias. They are mostly not focused on the issue of statistical bias, which is more or less beside the point to everyone, as LeCun and Gebru both pointed out.
I need to double check but from what I can recall, while she didn't mention any solutions or alternatives, she did point him to her previous work about the topic. (but she wasn't specific and I don't think she even included a link)
If you have diverse datasets, but your [prison recidivism, credit risk, or whatever] model doesn't like [giving light sentences to, issuing credit to] things correlated with being black... what then?
Or, for facial recognition, etc-- if you have done all the research with white faces, and then someone complains the system doesn't work for black people-- what then? Just saying "oh, with a different training dataset" to handwave things away doesn't mean that the resultant system will work well for minorities. The camera/optical system/etc may not like black people, too, and without additional validation you can't find these effects.
IMO, you don't need to know how to fix a problem where injustice is occurring to complain about and protest the injustice.
To expand a bit on your question directly and agree with your other points obliquely
> If you have diverse datasets, but your ... model doesn't like ... things correlated with being black... what then?
How do you release your data/models today? Does your release process include a section detailing what the data implies or the limitations of the mode? How would your boss handle being exposed to these findings? If your boss still wants to use the data set or deploy the model knowing the limitations, do you care enough to do something about it? Maybe you have a different set of answers to these questions. Does that make asking them any less important?
Hire more minorities in ML. One of the few industries where forced diversity is a good thing since your models will be better (wider range of opinions and sources).
See, it's not actually hard. Gebru is just correct in saying you are not listening.
Pretty diverse team if you ask me. Sprinkling underrepresented (and therefore 'underpowered') representation into engineering teams will solve a lot of issues, but I think it has yet to be proven that it's going to help ML bias.
You give too much credit to the people who build those systems. Having more minorities in ML is a good thing, but I don't think it's gonna change anything regarding bias in ML systems.
Yes, but that doesn't mean representative of all minorities. Very few Black people and Native Americans, for instance, even if Asian people are represented well.
This is immaterial. What about blind folks or quadriplegics or folks with chromosomal defects. Adding a black person to a team and expecting them to somehow fix anything beyond what they are trained at working on is unfair and unrealistic.
When the rationale is that having someone with an experience of being disadvantaged might be particularly inclined to help build a less biased system, one hardly refutes it by pointing out there are a lot of [not very disadvantaged] minorities on the team.
But one might earn some pedantry points with the argument.
Black cops shoot black men all day long. I don't see why black ML folks are going to be completely immune to all of the pressures that have allowed the domain to get to its current state. While I'm sure many would accept the challenge, it's waaay too much to ask or expect of them.
Yes absolutely this issue needs diverse perspectives and the priorities they bring, but it will fail without objective standards to ensure that technology in the field reliably meets our expectations.
> Yes absolutely this issue needs diverse perspectives and the priorities they bring, but it will fail without objective standards to ensure that technology in the field reliably meets our expectations.
It sounds like we mostly agree. It certainly doesn't hurt to have someone who's been the subject of unfair profiling before to get his spidey sense going at the implications of the system, but it isn't a cure-all.
Sure, and in some cases that does matter. But it seems unlikely that the PULSE model was biased towards white people because of latent racism, given that 3 of the 5 researchers who built it weren't white.
I wanted to write to address your point about the technical assertion, without getting into your request that Gebru empathize with one of the most prominent figures in science.
LeCun focused on the narrow technical question of statistical bias. Gebru and many many others have raised the point that an algorithm which works equally well for all skin tones will still have a societal impact, and they want to question that impact. How does a given new technology shift power? In whose favor, and at whose expense? That's the question which is arguably more important than some accuracy on a benchmark -- which as LeCun points out is easily solved with a more representative dataset.
Have you engaged with any of the critical study of AI, from Gebru or others, like the AI Now Institute at NYU or Ruha Benjamin at Princeton, or one of the lots of other scholars even in that one Twitter thread? The results of their work have included the recent IBM and Amazon moratoria on the sale of computer vision technology, and likely others to come.
The fact that LeCun refused to engage with a valid and deep criticism of a paper (which was not even his), to me at least, indicates a lack of willingness to engage critically with the social impacts of computer vision. Gebru's tweet pointed out that lack of willingness and her frustration with it, and also the fact that LeCun has had ample time to hear of it (it is in the news a lot, especially after these Amazon and IBM announcements, but groups like AI Now have been describing this for years.)
How about some empathy for Gebru? I would argue that she actually has more on the line than the single most celebrated individual in computer vision in this situation.
But that's the point, Lecun didn't reject the social impact, he was just not interested in that discussion. He was interested in a scientific discussion on the main cause of the bias. That was the all purpose of his first tweet. Then her reaction was about listening to marginalized people, which had nothing to do with the discussion - he never denied that those systems have bias and social impact.
>> But that's the point, Lecun didn't reject the social impact, he was just not interested in that discussion. He was interested in a scientific discussion on the main cause of the bias.
This isn't allowed anymore, for better or worse. If you don't talk about things with a social justice lens on Twitter, you're liable to get called out as ignorant. It's the #1 subject on the site right now, and no place for discussing anything remotely adjacent to it without invoking it.
Well yeah that's somewhat of a separate thing. If you say something but don't mention the right stuff, you get yelled at. If you are silent, you get yelled at.
In fairness, if you're gonna post a statement, you better go full bore with it with all the keywords in it. People were circulating rubrics on Twitter to grade social justice statements and brigading those who didn't hit all the notes.
Wait...ignoring a discussion on Twitter is a privilege now? Some people don't have the privilege to do so? Just...put down your phone? I guess that makes me super privileged, seeing as I don't visit that cess pool of a site.
Well, can you help me understand how the conversation was detoured in the first place? I tried to read it, but as mentioned I don't use Twitter so it's confusing to me.
If I read it right...a person posted a photo that turned pixelated Obama, a half white person, into a fully white person.
A person, who I assume is some ML expert, explained that it's because the training set used uses too many white faces.
A person replied very negatively to this answer, saying she's tired of it, that he doesn't listen, and talking about injustices in the world.
I don't think either of them are wrong, I just don't understand the leap. Turning pixelated Obama into some random white guy isn't a social injustice. I kinda felt like both weren't even participating in the same argument.
I think it does mean that other people can't be frustrated with that choice. It's very toxic behavior for me to jump into a conversation and tell people they're bad for not discussing other topics I find more important.
To be fair, LeCun started a discussion around bias in ML -- Gebru's reply is on topic, since statistical bias in ML is a narrow way of understanding the social impact of an algorithm. Gebru is one of the foremost scholars on that issue.
I can certainly empathize with Gebru. I also feel frustrated when people post bad takes on Twitter, especially when I know they know better.
That doesn't mean we should condone her decision to start yelling about it. Shouting matches are a bad thing, even when they take place on Twitter instead of a hallway - most people don't want to work in fields where disputes are resolved through shaming and caps lock.
If the problem is that the system is too efficient or too good (people need slack in their tethers —only catch 75% of tax cheats, deadbeat parents, etc.), I’m sure they can tweak the system to be “as efficient or accurate as people” but cheaper.
"The social impacts of computer vision" says more about the hype around the subject matter than anything. It's marketing, not science. Computer vision isn't that cool, yet. Running a deep convolutional network is a sales pitch by AWS to sell GPUs by the second.
I actually sell computer vision software - specifically designed to help reduce subjective evaluation of aesthetic material performance. Most computer vision is real prosaic. It's not scifi. You read a bunch of papers about horrible things done to marmot eyeballs in the 60s, build a discrete model because neural networks are notoriously hard to support in real-world applications where you have to explain yourself to a chemist, and move on. Her work is justifiably marginal.
Personally, I don't think that an interest in the social impacts of one's work as a researcher should be considered hype -- shouldn't one be interested in their impact on the world? For computer vision in particular, these systems are widely deployed. Also, they are just part of a larger apparatus of automatic decision systems that everyone now interacts with every single day, from getting your insurance rates to getting a loan to... Those have now become prosaic too -- certainly not scifi.
Indeed! She was the bully in that all debate. I also thought about narcissism when reading her tweets. It looked like tweets that someone like Trump could write.
He's right. Twitter is pretty toxic. I'm not sure if FB is better for such conversations and debates. But he's active there in case you want to continue following him.
I still don't get what solution they are proposing to the problem. I have also always assumed that any bias stems from data since the system uses it to learn. What's the alternative?
Yeah, I think that's what's missing for me too. Like, what do they expect LeCun to be doing instead?
I guess it's possible to suggest that, given the lack of diversity in training sets, different training techniques should be adopted to account for it. Is that actually a viable approach that people are suggesting, and LeCun is ignoring/downplaying?
Deciding what is fair and what is biased is a political question that impacts a lot of people who aren’t software engineers and it shouldn’t be up to them to decide.
I'm not from the US but there is a link between socioeconomic status and education. I would assume that with the large percentage of minorities, especially black Americans, raised in poverty that there would be a lack of suitable candidates. Just one of the ways that history of racism affects the modern world.
Do they have any practical suggestions on where to find a huge number of suitable qualified people? The people in prestigious academic positions often come from not just non-poverty backgrounds but wealthy backgrounds.
Facebook is rich enough to invest in black people's education and essentially train data scientists from the ground up. A black scholarship for data science would be a great gesture from them. Plenty of other industries do this for their fields as well.
Well educated people who can work on this problem are not minorities, i.e. they have access to education, higher social status, better starting point. While their color of skin might be different, they don't need to be aware of all the issues their race/sex faces. Example can be difference between treatment of certain minorities in US (where most of the science is developed) and the rest of the world. A scientist at Facebook/Google will develop something for the whole world having US centric view, without access to views from other cultures and their reqiurements for fairness.
If I had to summarize it in a sentence, the solution proposed is "consider the ethical implications of misapplication of your work, and attempt up front to mitigate the damage that would result." It's more nuanced than that, but it's the reason some people don't accept Yann LeCun's "just debias the inputs" as sufficient.
Also, consider that most of us here believe in the freedom to tinker, the freedom to use general purpose computing, and the freedom to freely share results.
These freedoms are also at stake here. "Banning bad AI research" is just an awful idea.
I recommend to all scientists that they disengage from Twitter. I know some think it's an effective way to communicate, but the vast majority of communication on the site seems to devalue rational thought. Ultimately, tweeting places your career at risk for reasons that are not well-aligned with reasonable principles.
One of the problems today is that you can be totally technically correct about something, but if you appear insensitive, you can be criticized to the point of being silenced. I am very troubled by the idea that rational, polite people who are trying to be helpful are being silenced because their point of view is not 100% consistent with that of the popular mob.
While I sympathize with this view: "You can’t just reduce harms caused by ML to dataset bias" (there's clearly other stuff like operator bias), I think there's a problem that if you don't focus first on dataset bias, you'll never reduce the harms caused by ML.
Maybe someone already said it, but I just don't see Twitter as a public space -- it is not Hyde Park Corner, it is a large courtyard, very large indeed, but privately owned. From this perspective, I am really not bothered of who is on it and who is not (I am not on it).
The topic discussed in the Twitter thread, on the other hand, is much more interesting. I am a layman in ML (although I have implemented a NN myself, but I am not a data scientist) so I offer my view hoping to get some feedback and be corrected. My view is that the original post was correct and LeCun was not. I just can't see what sort of objective function you can use to solve every potential problem arising in the future about similar issues. You want a "totally unbiased" model? Then, you use total diversity in your training set? Can't it happen that the model will then enforce diversity in a non-diverse picture? Is that your ideal result?
That might be a silly example and maybe not 100% correct for the example discussed, but, in general, my layman feeling is just that ML cannot help in cases where moral judgement is required. It can provide a good tool in any other case where, for example, a quadratic error is a reliable error measure, but that seems a hard limitation.
At this point, I think it would be a net win for society if all the tech companies got together and just ended Twitter. Revoke their DNS registration, shut down their cloud hosting, etc. Just completely end it. Twitter is tearing us apart. And I think it is intrinsically that way. Twitter is the meth lab of the of the Internet.
I hear this a lot, but I don't get it because the user is entirely responsible for who they follow and thus the messages they're exposed to.
I prune my Twitter feed and keep it pretty slim. I follow people who I find interesting. If they don't turn me off, I continue following them. If they do, I cut them. The turbulence in my Twitter feed remains moderate as a result.
You can always come back to someone, too, after a particular controversy has blown over or something. But my point is it's all in the user's hands how out-of-control Twitter feels. Simply don't follow people who make it miserable.
My concern with Twitter became reality when news media started using tweets as part of coverage.
Between APIs, bots and selective coverage it creates an ability to amplify messages as if they are representative of public opinion. Look at the services that will sell followers as an example.
I look at Twitter less than once a month but because of coverage (like the above article) I know about all sorts of stuff that’s happening on Twitter.
Twitter curates content and pretend to be neutral the way you said.
They push politics and constantly push you away from content you like. So everyone is disappointed and angry but thinks they enjoy it and some people think it’s all up to users.
I don't really understand sentiments like this - Twitter is just a means for people to communicate. If the apparent result of this communication is "tearing us apart", then isn't that the fault of the end to which people communicate, and not the means by which they communicate?
> I don't really understand sentiments like this - Twitter is just a means for people to communicate. If the apparent result of this communication is "tearing us apart", then isn't that the fault of the end to which people communicate, and not the means by which they communicate?
No, tooling and the incentives around them matter. Twitter makes it super easy to find the worst examples of the other side/take them out of context, and circulate what they said among 'your people' to get massive retweets. Twitter does this because it creates a more engaging experience. Yes, if people were perfect robots, Twitter could just be a means for people to communicate. But the incentive structure of social media, ie: what goes viral, selects for outrage. The medium is the message.
> Twitter makes it super easy to find the worst examples of the other side/take them out of context, and circulate what they said among 'your people' to get massive retweets. Twitter does this...
i mean, i can give plenty others that do exactly what you said. twitter is not unique and certainly not the problem.
i read his final(?) tweets and the reasons why he decided to leave twitter. and i don't think or remember him blaming the medium.
The pressure to keep one's communication brief due to the 240 character limit makes nuanced communication much more laborious, and that results in more conflict.
The medium is the message. The medium, Twitter, may not necessarily be just a passive channel, but an active component that amplifies and promotes certain types of communication at the expense of others.
Yes and no. Keep in mind that the algorithms that Facebook and Twitter use to increase engagement metrics might actually encourage conflict. As a result, the means of communication fundamentally change the framing of the resulting social interactions.
Now, within that system users can certainly be self-aware and actively abide to 'remembering the human.' Even then, this self-aware group is likely a minority.
> I don't really understand sentiments like this - Twitter is just a means for people to communicate
The problem is in the way it is designed. The shortness of the tweet limits context and nuance. This makes it easy to find something to be outraged over. Now add the re-tweet that makes it easy to spread the outrage everywhere. No, Twitter is optimized for outrage and conflict.
One perspective is that humans weren't meant to communicate so broadly and with such low costs. The result is the emergence of toxic dynamics that are usually held in check by the difficulty of communication, which raises the cost of signalling. When signalling is cheap, as it is on Twitter especially, then signals lose value in the same way that currency loses value in hyperinflation. People have to resort to more and more extreme signals in order to get a reaction from the network.
By making highly viral communication easier, Twitter has done something equivalent to removing control rods from a nuclear reaction. The reaction is fine when it's under control, but if it reaches a certain level of intensity, then different and highly damaging behaviors can emerge.
> Twitter is just a means for people to communicate
Human to human communication inherently includes a certain amount of noise per signal transmitted. Twitter or in general social media algorithms seem to amplify the noise more than the signal since disagreement translates to higher engagement metrics than agreement. There is only so much their algorithms can do in the name of increasing relavance (i.e finding signal amidst the giant noise they help created).
The problem isn’t social media, the problem is unmoderated social media that uses engagement metrics to drive their algorithms. They’re selecting for the most hateful, divisive content.
It’s YouTube, it’s Facebook, it’s twitter.
Those three sites are the problem. Reddit has a lot of hateful bullshit but you need to go out of your way to find it. YouTube and Facebook feed you a heroin drip of it all day long.
Interesting. I guess I take the term "social media" a little too literally. This is not a hill I'm willing to die on, though. You're most likely correct.
It always depends on how you stretch the term indeed, HN has less features than most old-style phpbb forums so that sounds strange to count it as social media.
How can such things have cults ? This was a fierece debate of ideas which should have been allowed to continue instead of people jumping in and spoiling the debate with personal attacks. Had this debate continued based purely on facts and best practices we would have had better ideas about bias for free. I just don't understand this behaviour. People are Yann LeCun "fans", people are PyTorch "fans" (such people also sent Francois Chollet hate mail), elon musk "fans" (if you bet against you are a bad person, wth). Just to be clear "fans" are not very smart people and just because someone likes one technology or one idea over another doesn't make them Yann LeCun or a great technologist or a pioneer. It just makes one an idiot. Great ideas should win or lose regardless of who said what.
What I’m reading is that researchers are getting mobbed for reporting their results. What are they supposed to do? Not report results? Report false results?
Then they go on to say this it can’t be solved by changing the dataset. Okay, exactly what are the researchers supposed to do, aside from give up and go into sociology?
In China it is enough not to speak against the CP, to be left alone and do your research. In the USA you actively have to signal your allegiance to the cause. And not only that, the cause can change overnight and what you said couple of years ago, may not be PC today.
There are two fundamentally distinct research questions at play:
1. Can a computer learn X at all (and how)?
2. Can a computer learn X in an unbiased way (and how)?
Where (1) is clearly a prerequisite for (2). Most of the ML community is currently focusing on (1), and Yann is right in saying that unbiasedness should not be a concern for those focusing on it. And Timnit is right in insisting that more focus should be spent on (2), which is a perfectly valid research question in its own right and not something to be left only to ML engineers.
With that said, not being familiar with Timnit's work and seeing her explode like that did not leave a good impression of her and her cause.
what I get from the tweets is that Yann offers a more or less technical explanation for a specific issue. then he gets screamed at with straw-man arguments and personal attacks by people who seem desperate for recognition in an emerging field (ethics of AI) (e.g. spamming about some single paper that they brought to "Yann's house", meaning an ML conference, as if it's the new Bible).
tbh Yann should've known what he gets into. everyone that knows anything about AI would already know the explanation. writing it on twitter would just attract an angry mob of pseudo-scientists.
I read with horror once here that in some places in the US, AI is used to grade pupils exams in the form of long-form essays.
AI in its current form should not be used ANYWHERE as the default arbiter where human accountability is needed. Like in grading an exam, or in dishing out a court sentence. That is a pretty obvious thing, but most people are immune to understanding obvious things, especially if it doesn't fit their agenda.
In fact, bias in statistical models has existed long before AI. For example, credit scores are biased, they will on average give lower score to a black person. So what is the politically correct solution here, to multiply credit scores by a certain factor based on race so average scores of all races are the same?
Apartment managers see credit score below X and reject an applicant. If they were to make their own decisions, it would be a legal liability. If they reject based on credit score and other non-protected criteria, they can't be sued.
Twitter is like Linkedin for self-destruction these days. It is not going to end well for the platform and its users. On the positive side, at least we know these ridiculous debates are happening, as opposed to someone simply posing a job for unspecified reasons
It may be a bit off-topic, but i'm still amazed people still consider twitter a good platform for expressing political views, or actually any views at all.
Instantaneous, short messages is clearly not the preferred medium for intelligent conversations.
I still see his account on Twitter so I am confused what does it mean to "quit" twitter? Can you not hide your account like you can in Facebook? Not familiar with the social media platforms so much.
I think part of the problem here is that she's concerned that the data is biased because the people training it are biased. The data isn't racist, researchers choosing data are racist.
Pointing out the data is biased is just pointing at a secondary cause rather than the primary cause. The humans are the location in the chain of causality (i.e. a learning human) where permanent change is possible.
It's a pun username. I don't think it's a misleading username. The poster actually refers to Bengio in the third person. And he makes some very good points.
In that Reddit thread the topic starter mentions that his CS faculty has a diversity issue. According to the Reddit TS because 30% of undergrads and 15% of the professors are women.
I was triggered by that and wanted to comment on it because I can imagine specific parts of CS are currently so new, evolving and hard to master that I can hardly imagine that diversity is considered an issue.
I can imagine society should already be happy with anyone who is able to complete an university CS study.These studies often require a specific mindset, passion and probably also a specific 'beta' brain.
I might be totally misunderstanding diversity anyway, but in my eyes all society can do is provide equal opportunities to everyone and the outcome can be a result of other factors.
In some specific industries like healthcare (specially nurses) and childcare the far majority is women. That is not because men don't have the opportunity to become one, but there are other factors that create this specific outcome where men are outnumbered.
In e.g. construction the majority is men. Also here women have the opportunity to get such a job, but women probably less likely apply for heavily physical construction jobs because of different physical conditions.
Therefore I think diversity itself should never be a goal, but making sure that _everyone_ gets equal chances/opportunities should be? And then accept the outcome.
In case of the Reddit TS it seems improvements on equal opportunities could perhaps be made on how is dealt with paternal leaves.
i wonder if the program should be ethical use of intelligence, not just artificial intelligence to prevent the reckless use of logic under an influence.
From the the perspective of a technical person, such as a software developer or AI scientist, it's easy to think about what we do as working with data and algorithms. While we might step back and look at a system as a whole, a lot of technical people can't communicate the birds eye view of what they're creating. They might have the capacity, but easier for them to rely on the technical description - yeah we can just make sure the data is good and the problem is fixed. However, that doesn't communicate fully and allow another person to understand the larger picture or that you even recognize it yourself.
As an example, consider the architectural view of a system where you need availability, performance, reliability, and scalability scalability. Over time we've learned to add Security. In recent years we've begun to learn how important how to include accessibility and privacy - we're learning and don't always get it right. All these things are often implied and we expect people to believe that we already know to include them. Now, lets look at more dimensions that are affecting what we build in the area of equality, diversity, and inclusion, which are part of our discussion, especially when we're talking about AI where we lack explainability.
So, it's true that we have to look at the data and that's part of the data science associated with AI work. When we are doing data science, it's more than just munging data to try to get a validation set match. We have to do the same thing we do with all other software and look at the domain of the problem, what it's purpose is, who are the users, and what are we trying to accomplish. If we examine the data set these things will help inform us on the appropriateness of the data being used, which takes analysis, just like other software problems.
It's simple to say, let's just make sure we have an equal amount of data from each represented group. While that isn't bad, it isn't enough. Imagine hiring or loan application program. We have demographics like name, location, occupation, sex, race, etc. Some obvious discriminants are sex and race. However, think about things like location - is it possible for someone to be denied a loan because they live in a part of town considered high-risk? Maybe a human wouldn't make that connection, but a machine learning algorithm with historical data that discriminates will also use that data to automate the discrimination.
We have to look at the history, current status, and goals of what we want to do to ensure we're thinking the problem through, rather than some mechanical input-process-output steps. That's why simplifying the discussion with "just change the data" doesn't communicate the source of the problem or how it should be fixed. It would have been nice for both people to ask "BTW, what did you mean by ...?" to open the conversation so we all could learn.
The state of AI fairness research is really poor, as in the focus is on politics and sincere effort to develop useful algorithmic notions of fairness aren’t even attempted.
It creates a way to take notoriety and clout within the AI field by doing work that is “about” fairness without actually solving fairness problems.
This article itself is a good example. Comparing Timnit Gebru’s body of work or authority in the field even remotely with Yann LeCun is preposterous, and Gebru happening to focus on the impacts of bias in the field doesn’t change that. Yet the article is totally loaded to highlight Gebru and undercut LeCun.
I think AI fairness is tremendously important for the world, but I don’t see anyone remotely tackling the problem in a rigorous way at all. Instead I see people trying to hype up fluff work about fairness as a hot topic into fame & clout so they can be de facto leaders in the overall field of AI despite not actually contributing to it (not even in fairness research).
The whole thing politicizes it in a really ugly way. It reminds me of the Feynman anecdote where people try to get him to settle a dispute about the physics of electricity having an implication for observers of the Sabbath.
“ It really was a disappointment. Here they are, slowly coming to life, only to better interpret the Talmud. Imagine! In modern times like this, guys are studying to go into society and do something--to be a rabbi--and the only way they think that science might be interesting is because their ancient, provincial, medieval problems are being confounded slightly by some new phenomena.”
Maybe we shouldn’t use Ai for anything in critical situations. Perhaps there is no perfect model for AI training. Nuance is valuable and perhaps current AI systems are not valid answers to the problems we face. Money will say otherwise but it’s hard to support these kinds of solutions. Better trained people are a solution that is unpopular currently.
Yann LeCun is an excellent engineer, and he is making pedagogy about AI. His point is totally valid, but yet, he gets political replys that lose him.
Yan LeCun would be comfortable talking about why back propagation in convNets does better with white tones and shadows, but is not a politician, seller, politically correct personality, able to answer some "You can't reduce the society bias of ML to some kinda data bias ALWAYS #somehashtag".
It is not good, when anyone who offers even a little bit of truth or reasonableness during a discussion is struck down as the enemy and attacked instantly by the vehemence of someone's offense.
If the successful chief scientist of a huge company can be taken down by forces such as this, while speaking truthful statements, what is the likelihood if I were a lowly junior researcher, I would dare voice an opinion or participate in the discussion? Or after a while, seeing what happens to such people, want to help?
Fear and exclusionary self-righteousness are not the way to win a social struggle. History shows that success comes from making more and more people feel they are a part of the solution -- not making them fear to open their mouths. Especially if, in the end, we each still stand alone in the voting booth and pull the lever based on how we've been made to feel.