Jacob Coxon 谈 AI 竞赛

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

今天我从 Anthropic 辞职了。在过去的三年里，我先后在 OpenAI 和 Anthropic 从事预训练研究。这两家公司行事都不负责任。它们正径直竞速冲向具备自我改进能力的超级智能，拿我们的生命做赌注。以下是我的更多想法。

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

不要低估这项技术的力量。这些系统很快就会具备超人能力，能够黑进任何系统、在一夜之间彻底变革任何领域，并获取真正的权力和资源。我们都亲眼见证了这些领域中的每一项进展，而且这种进展并没有放缓。

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

构建 AI 的人们真心认为，它可能会在本十年末杀死我们所有人。这绝非营销噱头。恰恰相反，许多高管和资深研究人员在面对媒体时会字斟句酌以显得合情合理——但我私下里听到过这些人表达恐惧。人类没有任何其他活动能带来如此程度的危险。

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

一种常见的反应是：“如果他们真这么想，为什么还要继续造它？”在 OpenAI，许多人尚未从深层次上内化这种关乎文明存亡的利害关系。而在 Anthropic，大家对此心知肚明，但他们却陷入了一场争先恐后的竞赛——他们认为没有其他人会负责任地行动，因此尽管冒着风险，他们也必须亲自抢先做到。

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

接受这场竞赛并步入“终局”，是一场傲慢的豪赌，绝不该由一家私营公司的 Slack 发起。试图在对齐上“速通”，必须建立在极度确信已无更好路径可选的前提之下。

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

我对协调合作的潜力持乐观态度。诸如 Hugging Face 遭受攻击之类的警示，使得美国各实验室之间达成步调协议变得更加可行。但我觉得我们目前还没有走上防止全球性竞赛的正轨，而要做到这一点，可能需要采取代价高昂的行动，例如暂时禁止提升模型能力。

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?

如果你是一名实验室研究人员，我敦促你想一想未来几年究竟会是什么感觉。你真的想在对其心智缺乏严谨理解的情况下，启动一次超级智能的强化学习训练运行吗？你应该因为“反正总会发生”而埋头默认——还是借此机会呼吁改变现状？