移动通信网 程然
这个夏天,围绕超级智能的争论从实验室内部走向了公众视野。Anthropic 研究员 Jacob Coxon 宣布辞职并警告称,他的前雇主与 OpenAI 正“径直冲向可自我改进的超级智能,并拿全人类的生命做赌注”。随后,Anthropic 对齐科学负责人 Evan Hubinger 回应称“我们确实真诚地相信 AI 可能杀死所有人类”,CEO Dario Amodei 也在接受 CNN 采访时表示“我与 Jacob 的共识远多于分歧”。当顶级 AI 公司的员工开始严肃讨论自家产品是否会导致人类灭绝时,一个更基础的问题却被相对忽略了:从工程角度看,超级智能(ASI)究竟是否可能实现?
递归自我改进:从思想实验到工程路线图
支撑 ASI 叙事的核心机制是递归自我改进(RSI):一旦工程师造出能够改进自身的 AI 系统,将触发自主升级的加速循环,最终催生认知能力远超人类的超级智能,而这样的系统可能轻易摆脱人类控制。这一概念并不新鲜,早在 1965 年,统计学家 I.J. Good 就在论文中推测 AI 自我改进的快速循环可能导致“智能爆炸”。这一想法在 1990 年代被湾区未来主义团体 Extropians 接纳,随后经由 Eliezer Yudkowsky 和 Nick Bostrom 等人物推广,逐渐渗入理性主义运动和有效利他主义社群。
变化在于,这一话题已从思想实验转变为实际工程目标。Anthropic 在今年 6 月发布的题为《When AI Builds Itself》的博客中描述,由大模型驱动的智能体正在加速代码编写、调试与优化,并自动化部分研究流程,其内部数据“显示 AI 已在加速 AI 系统的开发”。OpenAI 同样态度明确,在 9 月的报告中宣布目标是到 2028 年 3 月开发出自动化研究员。但乔治城大学一位计算机科学家在博士研讨课上对超级智能叙事的技术与文化根源进行系统梳理后,提出了几项关键质疑。
【关键数据】
- GPT-1 参数量(2018 年): 1.17 亿
- GPT-3 参数量: 1750 亿
- GPT-4 参数量(推测): 超过 1 万亿(较 GPT-1 增长近万倍)
- OpenAI 自动化研究员目标节点: 2028 年 3 月
- RSI 概念提出时间: 1965 年
![建议自制:GPT 系列模型参数量增长曲线图]()
规模扩张遇瓶颈:数据与算力的现实约束
质疑的核心之一指向训练数据的可获得性。回顾大模型的发展轨迹,2023 年之前的主要进步来自模型规模的扩张:参数量从 GPT-1 的 1.17 亿跃升至 GPT-2 的 15 亿,再到 GPT-3 的 1750 亿,直至 GPT-4 可能超过一万亿。然而 GPT-4 之后,持续扩大规模开始出现收益递减的迹象。工程师们因此将注意力转向“后训练”阶段,利用强化学习技术提升已训练模型在特定任务上的表现——通过奖励模型的正确响应,引导其在未来生成类似答案。
这条路线的问题在于,强化学习需要大量高质量的任务数据与人类反馈,而这些资源的供给远非无限。自我改进循环的前提是系统能够自主生成足够多、足够有信息量的训练信号,但目前的工程实践显示,无论在数据、算力还是评估方法层面,均存在难以绕过的瓶颈。换言之,从“AI 辅助写代码”到“AI 自主完成递归式自我升级”之间,横亘着一系列尚未解决的基础性工程难题,而这些问题很少出现在关于超级智能的公共讨论中。
比智能爆炸更现实的风险
对 ASI 叙事的审视并不等于对 AI 风险的否定。事实上,围绕超级智能的舆论场已呈现多元分化:有人认为 ASI 概念被用作营销工具,或被用来推动有利于头部实验室、挤压新竞争者的监管议程;有人专注于讨论 AI 设计超级病毒等具体灭绝场景的技术合理性;还有人主张对齐 ASI 在技术上可行,因此应当加速推进以迎接数字乌托邦。这些立场分歧巨大,却共享同一个未经充分验证的前提——超级智能在工程上是可行的。
这位计算机科学家的结论带有双重警示:即便承诺的智能爆炸永远不会到来,追逐它的过程本身也可能造成真实伤害——巨额算力与能源投入、行业竞争压力下的安全让步、以及公共资源向 speculative 目标的错配。从工程视角校准焦虑,把讨论焦点拉回到可验证的技术约束上,或许才是当前阶段对 AI 治理最有效的贡献。对于产业界而言,这意味着在“灭绝叙事”与“乌托邦叙事”之外,还需要第三条更务实的话语路径:基于具体能力边界的风险评估与治理框架。
出处:Why a Computer Scientist Thinks Superintelligence Won’t Happen
英文原文Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied , “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN , “I agree with Jacob much more than I disagree with him.” These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks. These ideas have prompted varied reactions. Some argue that the concept of ASI is being used for marketing or to instigate regulation that will help the incumbent labs squash new competitors. Others focus on the plausibility of such systems killing us all, debating the finer points of specific scenarios such as AI-engineered superviruses. Still others claim that aligning ASI to human interests is feasible and so we should rush ahead to usher in this digital utopia. Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible? The answer is important because it can help us calibrate our anxieties and develop the most effective possible responses to this emerging technology. I recently explored this topic in a computer science doctoral seminar that I taught at Georgetown University, in Washington, D.C., in which my students and I examined the technical and cultural roots of the superintelligence story. (For those who are interested, we published an annotated bibliography of our class readings.) I came away with several key objections to the feasibility of ASI, which I’ll detail below, as well as a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm. Objection #1: Recursive Self-Improvement Is Speculative RSI is an old concept. In a 1965 paper , statistician I.J. Good speculated that a rapid cycle of AI self-improvement could lead to an “intelligence explosion.” The idea gained traction in the 1990s among the Extropians , a Bay Area futurist group, where it offered a way to discuss ASI without having to address the engineering details of how one would build such a system. The concept was taken up by such notable figures as Eliezer Yudkowsky (who later founded the rationalism movement) and Nick Bostrom (who helped steer effective altruism to focus more on existential risks to humanity). Recently, however, this topic has shifted from thought experiment to practical engineering within both Anthropic (makers of Claude) and OpenAI (makers of ChatGPT). In a June blog post titled “When AI Builds Itself,” Anthropic described how LLM-powered agents are speeding up the writing, debugging, and optimization of code, and also automating parts of research such as running experiments and logging results. The company’s internal data, the report concluded, “is showing that AI is already accelerating the development of AI systems.” OpenAI has been similarly explicit about its intentions; in a September report, the company announced a goal of developing an automated researcher by March 2028. But do these efforts put the companies on a trajectory toward uncontrollable RSI? Several obstacles undermine this claim. Consider the availability of the data needed to improve AI systems. Through 2023, major advances in LLMs came from scaling up the size of the models. The original GPT-1 had 117 million parameters (values learned during training that shape how the neural network processes information). This quantity jumped to 1.5 billion for GPT-2, to 175 billion for GPT-3, to (likely) more than a trillion for GPT-4’s release three years ago. After GPT-4, however, continued scaling began yielding diminishing returns . This led engineers to increasingly turn their attention toward “post-training” phases, where techniques from reinforcement learning are applied to improve an already trained model’s performance on specific tasks. This approach rewards a model for producing successful responses, helping it better produce similar answers in the future. It requires, however, lots of