Hi Jeremy here - I wrote this article. The deeper I got into studying this, the more I realised the people writing the laws that will regulate AI actually don't really understand what they're regulating at all.
So I created this to try to at least help them create regulations that actually do what they think they're going to do.
California's SB 1047, which I analyse closely, currently totally fails to meet the goals that the bill authors have stated. Hopefully this will help them fix these problems. If you have views on SB 1047, you can make a public comment here: https://calegislation.lc.ca.gov/Advocates/
Let me know if you have any questions or comments.
Jeremy -- thank you for sharing this on HN. And thank you also for everything else you've done for the community :-)
I agree with you that one of the biggest issues -- maybe the biggest issue -- with the proposed legislation is that it fails to differentiate between "releasing" and "deploying" a model. For example, Jamba was released by AI21, and Llama3 was released by Meta. In contrast, GPT-4o was deployed by OpenAI and the latest/largest Gemini was deployed by Google, but neither model was ever released! We don't want legislation that prevents researchers from releasing new models, because releases are critical to scientific progress.
However, I'm not so sure that lack of understanding by politicians is the main driver of misguided legislation. My understanding is that politicians prefer to consider the opinion of "experts" with the most "impressive" pedigrees, like high-ranking employees of dominant tech companies, most of which don't want anyone to release models.
Interestingly enough, in this case none of the 3 sponsoring organizations are dominant tech companies. Rather, they are well-funded AI safety orgs -- although, to be fair, these orgs generally get their money from folks that are some of the biggest investors in Anthropic, OpenAI, and Google.
But I do have the impression they earnestly believe they're doing the right thing. The AI safety orgs have done incredibly effective outreach at universities for years, and have as a result gotten a huge amount of traction amongst enthusiastic young people.
Thank you. Like you, I also have the impression that folks at those AI safety orgs sincerely believe they're doing the right thing.
But I would ask them the same questions I would ask the politicians: Which "experts" did they consult to reach their conclusions? From whom did their "talking points" come?
I hope you're right that they and the politicians are willing to change course.
Jeremy -- this is interesting and worthwhile. Thank you!
In the same spirit (ignoring the question of whether this sort of attempted regulation is a good idea), I have a question:
Debating release vs. deploy seems a bit like regulating e.g. explosives by saying "you can build the bomb, you just aren't allowed to detonate it". Regulation often addresses the creation of something dangerous, not just the usage of it.
Did you consider an option to somehow push the safety burden into the training phase? E.g. "you cannot train a model such that at any point the following safety criteria are not met." I don't know enough about how the training works to understand whether that's even possible -- but solving it 'upstream' makes more intuitive sense to me than saying "you can build and distribute the dangerous box, but no one is allowed to plug it in".
(Possibly irrelevant disclosure: I worked with Jeremy years ago and he is much smarter than me!)
Yes I considered that option, but it's mathematically impossible. There's no way to make it so that a general purpose learned mathematical function can't be tweaked downstream to do whatever someone chooses.
So in that sense it's more like the behaviour of the pen and paper, or a printing press, than explosives. You can't force a pen manufacturer to only sell pens that can't be used to write blackmail, for instance. They simply wouldn't be able to comply, and so such a regulation would effectively ban pens. (Of course, there's also lots of ways in which these technologies are different to AI -- I'm not making a general analogy here, just an analogy to show why this particular approach to regulation is impossible.)
I would not say it’s impossible… my lab is working on this (https://arxiv.org/abs/2405.14577) and though it’s far from mature - in theory some kind of resistance to downstream training isn’t impossible. I think under classical statistical learning theory you would predict it’s impossible with unlimited training data and budget for searching for models but we don’t have those same gaurentees with deep neural networks.
That makes sense. Regulating deployment may simply be the only option available -- literally no other mechanic (besides banning releasing models altogether) is on the menu.
Jeremy, this is a great read, thank you! What do you think about the amended version from today that gives the FMD significantly greater authority on what would and would not be covered by the legislation? Any other specific recommendations for the legislation that would help protect open-source?
Edit to ask: Does it seem likely to you that one of the unintended consequences of the legislation as written is that companies like Meta will no longer open-source their models?
It'll take me a while to digest today's changes, so I don't have an educated opinion about them as yet.
Yes, companies like Meta will no longer be able to open-source their models, once their compute reaches the threshold, if the bill goes through and if the "covered model" definition is interpreted to include base models (or is modified to make that clear).
this belongs in major newspapers and media outlets. some PR campaign to get this message out there should prove fruitful. you can hire someone and just have them suggest articles to magazines and papers, who are always looking for content anyway. it's topical, urgent, convincing, and it comes from an authority in the field, so it checks all the boxes IMHO
There is no agreed definition of what an open source AI model is.
I guess if the models you mention would have to be packaged in Debian, they would end up in the non-free section, since you cannot rebuild them from the training data, which is not published.
So I created this to try to at least help them create regulations that actually do what they think they're going to do.
California's SB 1047, which I analyse closely, currently totally fails to meet the goals that the bill authors have stated. Hopefully this will help them fix these problems. If you have views on SB 1047, you can make a public comment here: https://calegislation.lc.ca.gov/Advocates/
Let me know if you have any questions or comments.