Asterisk/Zvi on California's AI Bill - by Scott Alexander
California’s state senate is considering SB1047, a bill to regulate AI. Since OpenAI, Anthropic, Google, and Meta are all in California, this would affect most of the industry.
If the California state senate passed a bill saying that the sky was blue, I would start considering whether it might be green, or colorless, or maybe not exist at all. And people on Twitter have been saying that this bill would ban open-source AI - no, all AI! - no, all technology more complicated than a toaster! So I started out skeptical.
The bill applies to “frontier models” trained on > 10^26 FLOPs - in other words, models a bit bigger than any that currently exist. GPT-4 doesn’t qualify, but GPT-5 probably will. It also covers any model equivalent to these, ie anything that uses clever new technology to be as intelligent as a current 10^26 FLOPs model without actually using that much compute. It places three 1 types of regulation on these models:
First, companies have to train and run them in a secure environment where “advanced persistent threats” (eg China) can’t easily hack in and steal them 2 .
