We had rather grand expectations for the arrival of self-improving artificial intelligence. A machine would announce its awakening. Screens would flicker. Somewhere, a scientist would remove their glasses very slowly.
Instead, we have an amendment to a securities filing.
On 10 November, Anthropic told prospective investors that it had determined it had met a threshold in its Responsible Scaling Policy concerning AI-accelerated research. The crucial qualification came in the same paragraph: the determination rested on expectations about future models and the overall pace of progress, rather than an observed crossing by current models. The company was still assessing the underlying data. Civilisation’s next chapter apparently requires close reading of the prospectus. Anthropic’s amended filing
That qualification matters. A safety threshold is a rule for deciding when to take precautions. A demonstrated capability is an empirical finding. They can overlap, but they answer different questions. Sensible precautions often begin before the evidence becomes conclusive; waiting until the house has achieved full combustion is an unusually rigorous approach to fire safety.
Under the acceleration route in Anthropic’s policy, the relevant threshold involves observing or expecting roughly twice the rate of progress in aggregate AI capabilities against specified baselines, with a substantial contribution plausibly coming from automated research and engineering. This concerns the pace at which capabilities improve. Doubling the number of lines of code produced before lunch would establish something considerably less impressive, particularly if half of them subsequently need deleting. Responsible Scaling Policy, version 3.4
The threshold deserves attention. It also deserves its footnotes. Removing them makes the story more exciting in much the same way that removing the denominator improves almost any statistic.
The underlying idea, recursive self-improvement, is straightforward. AI helps develop a more capable successor. That successor becomes better at developing the next one. Improvements feed back into the process that produces improvements.
The loop can run through an entire laboratory: models, software tools, experiments, training infrastructure and people. It need not involve a solitary chatbot staring into its own source code until it achieves enlightenment. Anthropic describes a progression towards systems that could autonomously build their successors, while distinguishing that possibility from its current capabilities. Its published account reports strong performance on executing research tasks alongside persistent weaknesses in choosing which goals to pursue. Anthropic’s account of AI-assisted development
That distinction is familiar to anyone who has worked in research. Doing the experiment is one problem. Working out whether the experiment deserves to exist is another. Universities have developed an elaborate system for managing this distinction, principally involving meetings.
Dwarkesh Patel’s recent argument centres on verifiability. Mathematics and programming provide opportunities to check results and learn from repeated attempts. He argues that enough of AI research has similar properties for automated researchers to become exceptionally capable: experiments can reveal whether a change improves performance or efficiency. This is a forecast about how far those feedback mechanisms will carry us. Patel’s essay on automated AI research
There is an appealingly unromantic possibility here. Some discoveries may require less of the lone genius gazing through a window and more of the patient investigator trying another combination, measuring the result and keeping proper notes. Humanity has always respected persistence. We have simply preferred other people to perform it.
Yet a reliable answer to a narrow question leaves the larger question open. An experiment can show that a model performs better on a chosen test. Establishing that the test captures what matters takes additional work. Accelerating the production of answers also increases the importance of asking useful questions.
A laboratory that becomes extraordinarily efficient at optimising an inadequate measure has achieved a very expensive form of tidiness.
There is nevertheless evidence worth taking seriously. METR’s November assessment estimated that Anthropic’s internal benchmark scores had been improving at approximately 1.9 times their earlier rate. Its broader judgment was that AI assistance probably accounted for most of the recent acceleration, with considerable uncertainty. It also reported that assistance remained more important than fully autonomous contributions. Anthropic redacted much of the evidence METR considered most informative, limiting the public’s ability to assess the conclusions independently. METR’s assessment
Those findings support concern about acceleration. They leave substantial room for disagreement about its magnitude, causes and continuation.
The redactions create a particularly modern predicament: we are being asked to prepare for a potentially extraordinary development using evidence that, in significant part, we cannot inspect. Commercial confidentiality may explain that situation. It does little to make public judgment easier. A black rectangle can conceal a trade secret; it cannot explain one.
From here, the discussion branches into two especially memorable possibilities.
In Patel’s spark scenario, automating research produces a tremendous burst: perhaps three to six years of progress compressed into one. Harder problems and resource constraints then restrain the acceleration. In his software-only singularity scenario, improvements to automated researchers outpace the increasing difficulty of further discoveries, driving rapid successive advances even without additional compute, until existing hardware imposes limits. These are conditional scenarios, with uncertain assumptions about how research scales. Patel’s discussion of spark and singularity
Calling the first possibility a spark is admirably restrained. Several years of technological change arriving within a single budget cycle would give the average institution just enough time to establish a working group with the wrong terms of reference.
Consider what such compression could feel like. A professional begins retraining for a newly valuable skill. Halfway through the course, the tools change. A company restructures around capabilities that become ordinary before the restructuring finishes. A regulator completes a consultation on systems whose successors are already raising different questions.
No infinite intelligence is required for this to become uncomfortable. A finite amount of progress can cause considerable disruption when delivered faster than people can adapt to it.
The benefits could arrive with the same compression. More useful experiments, better scientific software and cheaper technical work could expand what small teams can attempt. Problems abandoned because investigating them took too much time might become practical again. The attraction of automated research is easy to understand: there is an enormous amount of useful work waiting behind the sentence “we do not have the resources.”
The challenge is that faster discovery and faster absorption are separate achievements. A promising medical result still needs validation. An improved industrial process still needs equipment. Institutions still have to decide who can use new capabilities, who pays, and who bears the consequences when something fails.
Even the singularity scenario retains an awkward relationship with the physical world. Software can change faster than factories can be built. An ingenious design still needs materials, energy and manufacturing capacity. Intelligence may find ways around particular constraints, but each proposed workaround has to survive contact with reality.
Reality has an excellent record of declining invitations to scale.
The important uncertainty is how much useful improvement remains available inside the existing system. Could better algorithms unlock extraordinary capabilities from today’s hardware? Would each advance make the next easier, or expose a more difficult obstacle? How much apparent progress would survive broader evaluation?
Attaching a dramatic name to the process does not settle those questions. Neither does extending a curve until it reaches the top edge of the slide. Extrapolation is useful; the universe has yet to agree to its terms and conditions.
Meanwhile, meaningful human control becomes a practical design problem.
Imagine a research process generating proposals faster than its supervisors can understand them. Keeping a person responsible for approval may preserve the organisational chart while gradually emptying the role of substance. A signature can establish accountability without establishing comprehension.
The human becomes the person who clicks Approve because the other option requires three weeks of study and the next experiment is ready in forty seconds.
Avoiding that outcome would require ways to inspect results, challenge assumptions, reproduce important findings and stop work when confidence is inadequate. AI could help with those tasks too. But an assurance system needs some way to detect shared mistakes; otherwise, several models agreeing may amount to one misunderstanding with excellent attendance.
This is where the threshold story becomes interesting beyond the headline. If research is accelerating, the ability to understand and govern it must keep pace. Better measurements, clearer disclosures and credible review become part of the engineering problem. They are also part of the public bargain, because the consequences will extend well beyond the laboratories.
We can take the possibility of rapid self-improvement seriously while keeping our forecasts provisional. A spark would already demand substantial adaptation. A faster, sustained feedback loop would demand more. Both justify careful work before the outcome becomes obvious.
For now, the singularity has filed its paperwork. Someone should read the footnotes before we give it signing authority.




No comments yet