Outside Audit Finds Anthropic Is Building AI Twice as Fast. Why Independent Checks Matter

Photo: Anne Nygård / Unsplash

An independent research group says one of the world’s leading AI companies is now improving its systems about twice as fast as it did a year ago, and that AI itself is a big reason why. The analysis, released earlier this month by the Berkeley, California, nonprofit Model Evaluation and Threat Research, known as METR, looked inside Anthropic as the company prepares for a stock market debut planned for this month. Associated Press reporting on the findings highlights two things everyday users should understand: AI tools are changing faster than many people realize, and outside checks on the companies that build them are still rare. Here is what the report found, why it matters, and how to keep up.

What METR found

Anthropic invited METR to review its internal data and interview staff. According to AP, the group’s main findings were:

• Roughly double the pace. METR estimated that Anthropic’s models are improving on well-defined tasks about twice as fast as they were a year earlier.
• AI is speeding up AI. The researchers believe AI tools are a major reason for the faster progress, as the company increasingly uses its own systems to help build newer ones.
• Not just more computing power. METR said it could confidently rule out growth in training compute as the explanation, and found no bottlenecks, such as shortages of chips or data, holding the company back.
• Deep access. The analysis drew on about 11 weeks of internal data, from mid-August through late October, along with employee interviews. METR said it had “unprecedented access to internal data” on the company’s research and development.

METR also warned that using AI to improve AI could cause the acceleration to compound, and said Anthropic researchers described early signs suggesting AI could soon make bigger contributions to training new models.

The caveats matter

The report was careful about its limits, and those limits are worth knowing before drawing big conclusions.

• The speedup may apply mainly to tasks that can be graded automatically, not to messier, real-world work where success is harder to measure.
• Anthropic’s redactions limited how precisely METR could describe the acceleration.
• METR’s own team had internal disagreements and uncertainty about the findings.

It is also worth remembering who asked for the review. Anthropic invited METR in, chose what data to share and redacted parts of the results. That does not make the findings wrong, but it does mean this was a cooperative review rather than an adversarial audit. A truly complete picture would require repeat studies, comparable access at other companies, and clear standards for what outside reviewers are allowed to see and publish.

In other words, this is a significant signal, not a final verdict. It suggests the pace of change is rising at one major lab, measured in a specific way, over a limited period.

Why it matters to everyday users and businesses

If the tools you rely on are improving twice as fast, that has practical consequences. Features will arrive more quickly. Habits and workflows that worked six months ago may be outdated. Advice about what AI can and cannot do will go stale faster. And because companies are racing, they may change products, prices and terms more often.

The report also connects to the company’s own warnings. In its IPO filing, Anthropic told investors that AI’s growing role in its research could push progress beyond its ability to evaluate and safeguard its models. METR’s analysis offers outside evidence that the dynamic the company described is real, at least for certain kinds of tasks.

That is why independent review matters. AP noted that AI companies, encouraged by the Trump administration, have largely been policing their own safety practices. Findings from an outside group with real access are one of the few ways the public can get a check on what is happening inside these companies.

The backstory: a CEO who asked the industry to slow down

The report did not come out of nowhere. In a September 12 essay, Anthropic CEO Dario Amodei urged the AI industry to slow the pace at which it improves the capabilities of its models. He warned that recursive self-improvement, AI systems helping to build better AI systems, could outrun human control if left unchecked. He specifically named METR as an appropriate third-party evaluator.

OpenAI CEO Sam Altman said at the time that he also supported independent evaluators having employee-like access to AI companies. METR has evaluated models for Anthropic, OpenAI and other labs for years. According to AP, it is funded by donations and does not accept payment for its work, though partner companies provide access and computing credits for testing.

The tension is obvious. The same company calling for caution is also moving fast, preparing what could be the largest IPO ever, and reporting that its own progress is accelerating. Inviting an outside group in to measure that progress, and letting the results be published, is one way of trying to show that speed and accountability can coexist. Whether that model spreads across the industry is one of the most important questions in AI right now.

How to keep up when AI changes this fast

For everyday users

1. Re-test your assumptions every few months. If you decided a tool was not good enough for a task, try it again. If you trust a tool for something important, spot-check it again too.
2. Read update notes. When an app announces a new model or feature, take a minute to see what changed, especially around privacy settings and what the tool can do on your behalf.
3. Keep verifying. Better models still make mistakes. For health, legal and money questions, confirm with a qualified professional.
4. Be thoughtful with agents. Tools that act for you, sending messages or making purchases, should start with small tasks and clear limits.

For small businesses

1. Schedule regular reviews. Put a quarterly check on your calendar to review which AI tools you use, what they cost, and whether newer options do the job better.
2. Ask about outside testing. When choosing a vendor, ask whether independent groups evaluate its models and whether results are published.
3. Document your processes. Write down how AI fits into each workflow, so you can swap tools or roll back changes without disruption.
4. Train people, not just systems. Staff who understand how to check AI output are your best protection as tools change quickly.

Questions worth asking any AI company

• Who outside your company tests your systems, and what access do they get?
• How do you tell customers when a model’s behavior changes significantly?
• What safeguards apply when your tools take actions rather than just answer questions?

What to watch next

• Anthropic’s IPO. Once public, the company will file regular reports, giving outsiders a steadier view of its risks and progress.
• Other labs. Watch whether OpenAI, Google and others invite similar outside reviews with comparable access.
• Policy responses. With the federal government largely leaving safety to the companies, states and Congress may debate whether independent evaluations should be required.
• Follow-up measurements. A single 11-week snapshot is a starting point. Repeat studies will show whether the acceleration holds, grows or fades.

The bottom line

The AI in your apps is not standing still, and according to this outside analysis, it may be speeding up. That is exciting and unsettling at once. The practical response is steady attention: revisit your tools often, keep humans responsible for important decisions, and favor companies willing to let independent experts look under the hood.

Source: The Associated Press. Original AP report on the METR analysis republished by ABC News.