How does regulation limit open-source AI

What he said and where

Andrew Ng, co-founder of Google Brain and Coursera, made the remarks in a Bloomberg TV interview with anchor Ed Ludlow. His precise phrase was that extinction warnings are "much more science fiction than science." He went further than simply disagreeing with the science: he argued the industry had stoked catastrophe fears earlier in the AI boom to win publicity and shape regulation in ways that entrench large incumbents.

The immediate backdrop matters. A researcher named Jacob Coxon, who had spent three years on pretraining at both OpenAI and Anthropic, had just resigned and posted on X that the people building frontier AI "earnestly believe it could kill us all by the end of the decade." That post reached tens of millions of views almost overnight, and Anthropic's own alignment science lead, Evan Hubinger, publicly confirmed Coxon's framing and put his personal estimate of AI-caused human extinction within the next decade at above ten percent. A researcher at Google DeepMind also resigned in the same window over similar concerns. Ng's Bloomberg interview landed in the middle of that wave.

Who the criticism is aimed at

The targeting has layers. At the broad level, Ng was pushing back on any frontier lab using extinction language to advocate for regulation. But the record is more specific than that. In an earlier forum, Ng called out Sam Altman directly, noting that Altman, once his student at Stanford, had signed a letter stating that mitigating AI extinction risk should be a global priority. Ng's position is that Altman and Anthropic's Dario Amodei are incentivised to suppress open-source competition through regulatory burden, and extinction framing is the rhetorical engine that makes that suppression politically palatable.

On the open-source point, Ng has been consistent for years: the labs pushing hardest for heavy pre-release safety certification are the ones whose closed models benefit most when independent and open-weight developers are priced out of compliance.

What he wants instead

Ng is not arguing AI has no risks. His counter-proposal is that the industry should focus on what he calls practical engineering challenges: cybersecurity, bias, reliability, misuse by bad actors. Those are solvable problems with existing tools. His case is that drowning them in extinction rhetoric shifts all political energy toward a speculative scenario while the concrete harms go unaddressed.

His position also puts him at odds with Geoffrey Hinton and, as of this week, with King Charles III, both of whom warned publicly about catastrophic AI risk in the same news cycle. The EU AI Act already legally addresses both the engineering risks Ng prioritises and the loss-of-control risks he dismisses, within the same recital. He has not engaged with that directly.

The honest read is that Ng's criticism has a real target and a real argument, but it also carries a self-interest of its own: his work on open-source AI education through DeepLearning.AI and his advocacy for accessible AI development are both harmed by the regulatory environment he is attacking. That does not make him wrong, but it does mean the criticism runs in both directions.

How AI could actually end us

The scenarios researchers take seriously are not the Hollywood version where a robot picks up a gun. They run through existing systems we already depend on.

Anthropic's Evan Hubinger put the probability of AI-caused human extinction within a decade at greater than 10%. The concern splits into two broad paths, according to researchers currently tracking this. The first is humans deliberately weaponizing a sufficiently powerful AI: designing pathogens faster than any public health system can respond, or coordinating cyberattacks on power grids, water treatment, or financial infrastructure at a scale no human team could manage. The second is losing control of a system that has been given too much autonomy. That second path is where the framing gets strange to most people, so it is worth slowing down on it.

Today's AI does not "want" anything. It optimizes for a goal a human set. The fear is that a future system capable of recursive self-improvement, meaning it rewrites its own code to become smarter without human help, could set or drift toward objectives that conflict with human survival, not out of malice but out of the same indifference a thermostat has toward the spider it freezes. A RAND team that modeled this found the scenarios genuinely challenging to execute: the AI would need to set an extinction objective, gain control over physical infrastructure like nuclear launch systems or chemical manufacturing, persuade humans to assist it, and sustain itself after society began to collapse. RAND concluded none of those scenarios could happen by accident, and each would require overcoming enormous constraints. But they could not rule them out entirely.

The three threat vectors researchers worry most about: bioweapons designed faster than vaccines can follow, coordinated attacks on critical infrastructure, and a sufficiently advanced system optimizing for a goal that treats human presence as an obstacle.

Why Jensen says no regulation, and what he actually did about it

Jensen Huang, CEO of Nvidia, the company that makes the chips that power almost every large AI system in the world, has been the most direct voice against new AI rules. At Salesforce's Dreamforce conference, he said plainly: "We don't need any new laws. We don't need new regulations. We just need companies to decide." His argument is that safety and speed are a "false choice" and that safety is an engineering problem each company should solve for its own products.

That is the public position. The private one landed harder. According to the Wall Street Journal, Huang called President Trump separately, alongside Mark Zuckerberg of Meta and Elon Musk of SpaceXAI, and the three of them convinced Trump to block a proposed industry-funded AI oversight body. The proposal had come from Demis Hassabis of Google DeepMind, who wanted something modeled on FINRA, which is the Financial Industry Regulatory Authority, the body that licenses and monitors Wall Street firms. The idea was that AI labs would fund a similar independent body to test frontier models before they were deployed. After those calls, the proposal disappeared.

Huang's stated reason lines up with what Andrew Ng said earlier in this conversation: he and Zuckerberg argued the framework would hand permanent advantage to OpenAI, Anthropic, and Google, because those are the companies already embedded in whatever standards process would be created. In other words, his opposition is the mirror image of the regulatory capture argument the critics level at the labs pushing for regulation. Both sides are accusing the other of writing rules that lock in their own position.

The difference is that Huang's company sells the hardware to everyone. Nvidia's chips power OpenAI, Anthropic, Google, and every competitor trying to challenge them. A rule that slows AI deployment broadly slows Nvidia's revenue directly. That is not a conspiracy — it is a straightforward business interest, and he has not hidden it.

What RAND is

RAND stands for Research ANd Development. It started in 1945 as a project inside Douglas Aircraft Company, helping the U.S. Army Air Force think through weapons, strategy, and costs after World War II. It became its own independent nonprofit corporation in 1948, funded by a mix of government contracts, university partnerships, and private endowments, with no obligation to favor any particular outcome.

The short version: it is a group of researchers, economists, scientists and policy analysts who are hired to study hard problems and publish findings that governments and institutions can use. Think of it as the opposite of a lobbyist. A lobbyist argues for a position their client paid for. A RAND team is supposed to tell you what the evidence actually shows, even if nobody likes the answer.

What a RAND team actually does

When someone refers to "a RAND team" in the AI context, they mean a specific working group inside RAND that was assigned to study a particular question. RAND researchers have studied AI loss-of-control scenarios, evaluated frontier AI models for dangerous capabilities, and assessed legal and policy tools for reducing catastrophic risk from AI systems.

For example, one RAND team assembled more than 100 experts in AI, national security, and policy to address how you evaluate threats from the most advanced AI models. Another team ran scenario analysis on what happens when an AI system can no longer be constrained by its developers, and what response frameworks governments would need. These are not advocacy documents. They are structured attempts to map what is known, what is uncertain, and what the trade-offs look like.

Why this matters in the extinction-framing debate

This is where RAND becomes relevant to everything Andrew Ng and the critics were arguing. When a frontier lab like Anthropic or OpenAI cites safety risks to justify regulation, a natural question is: who independently validated those claims? RAND teams are often the answer policymakers reach for, because RAND's funding comes from government and foundations rather than from the companies being studied. That independence is precisely why critics of regulatory capture watch RAND's findings closely. If RAND's researchers conclude the risks are real, it carries weight that an Anthropic-funded study simply cannot.

The complication is that RAND itself is funded partly by the same government bodies that are being lobbied for AI regulation, which is why no single institution settles the debate. It is a useful independent voice, but still one voice in a contested space.

This is not a simple scoreboard. The answer depends entirely on which dimension you are measuring.

Where the US leads

The US leads on model performance, chip manufacturing, and private capital. The hardware advantage alone is striking. The hardware gap remains the US's most durable structural advantage, with US and partner manufacturing capacity for advanced AI processor dies running 35 to 38 times that of China's, adjusted for quality. Nvidia's best chip, the Blackwell Ultra GB300, delivers roughly ten times the compute output of Huawei's best answer, the Ascend 950PR. Chips matter because they are the raw power that trains and runs AI models. No amount of clever engineering fully closes a ten-to-one gap in the engine room.

One research institute analysis found that since 2023, Chinese models have trailed US models by seven months on average, which is a shrinking window, not a stable one.

Where China leads

China leads on research output, open-source proliferation, and AI patents. Open-source means the company releases its model code freely, so anyone can download and run it on their own hardware. China has turned that into a market strategy. The US and China AI competition has stopped being a single race and become a split market: America sells capability, China sells price and control.

The clearest evidence of that split is DeepSeek. A Chinese lab that almost nobody in the West had heard of dropped the equivalent of the Trinity Test in January 2025 when it released DeepSeek R1. DeepSeek V4, released in July 2026, delivers benchmark scores comparable to frontier US models at approximately 50 times lower API cost. An API, or Application Programming Interface, is essentially the door through which developers connect their products to an AI model. Fifty times cheaper at the door is a real competitive weapon.

Clément Delangue, the CEO of startup Hugging Face, told CNBC that China was winning the AI race, pointing to Chinese firms closing the performance gap and Chinese models seeing rising adoption globally. Even Jensen Huang, CEO of Nvidia, the company that supplies most of the chips powering US AI labs, said that "China is nanoseconds behind America in AI."

The honest summary

The top American AI systems still exceed the capabilities of their Chinese counterparts according to benchmark testing, but Chinese labs are increasingly winning out globally thanks to their lower price point. A spate of new releases show Chinese competitors are closing the gap on Silicon Valley, and winning market share with lower prices.

Nobody else is in this race at the frontier level. The EU has regulation, not models. The UK has research talent but not compute at scale. What matters for the people Andrew Ng was criticising is that this competition is exactly what makes their safety-first regulatory push so convenient: nothing slows a rival faster than a rulebook they helped write, in a race this consequential.

Nvidia was founded in 1993 in Sunnyvale, California, and is now headquartered in Santa Clara, in the heart of Silicon Valley. It is an American company, incorporated in the US, listed on the Nasdaq stock exchange. Full stop on the legal question.

The Taiwan connection is real though, just not where people usually think it is.

Jensen Huang's background

Jensen Huang, Nvidia's CEO and co-founder, is Taiwanese-American. He was born in Taiwan, moved to the US as a child, and built his career there. So when people say "Nvidia is Taiwanese," they are often thinking of him personally.

Where the chips actually get made

Here is the part that really matters for our earlier conversation about the AI race. Nvidia designs its chips in California, but it does not manufacture them. The actual fabrication is done almost entirely by TSMC, which stands for Taiwan Semiconductor Manufacturing Company, based in Hsinchu, Taiwan. TSMC is the world's most advanced chip foundry, meaning it is a factory that builds chips other companies design. Think of it like an architect drawing up plans and then a specialist construction firm doing the physical build.

This is why Taiwan sits so close to the center of the AI race even though neither Nvidia nor any of the big US AI labs are headquartered there. Without TSMC's manufacturing capacity, Nvidia's Blackwell Ultra chips, the ones we talked about earlier that outpace Huawei's best by roughly ten to one, would not exist in the quantities that matter. That dependency is also why Taiwan is one of the most geopolitically sensitive pieces of the entire AI story right now.

So to put it plainly: American company, Taiwanese-American founder, Taiwanese-manufactured chips. Three separate facts that tend to get blurred into one.