A company knows more than the public about how it trains and uses its models. An outside evaluator can narrow that information gap if it can ask its own questions and explain what it actually observed. That is the issue behind Anthropic’s arrangement, whose terms were clarified on 16 October.

Seeing more does not mean seeing everything

Anthropic describes providing hardware, workspace and internal access. Certain categories of information remain excluded or restricted, including customer and architecture information. Describing access as unlimited across all data would therefore mislead.

For readers of a future report, this distinction matters. A conclusion makes sense only within its scope: models examined, situations observed and unavailable information. Not finding a problem in one test does not prove it cannot occur elsewhere.

Publication is part of oversight

Under the announced terms, Anthropic has no general veto over METR’s publication, but may require redaction of information in predefined categories. The company determines how those categories apply; METR retains certain opportunities to comment on access received. This qualification must accompany descriptions of the agreement.

Our analysis: independence also depends on the ability to make the work’s limitations visible. A useful report can distinguish observation, interpretation and an open question rather than simply offer a reassuring label.

Results to await, not invent

METR already publishes evaluation work. Its website distinguishes partnerships from evaluations without the relevant company’s involvement. That context does not predict conclusions from the newly announced access at Anthropic.

For buyers of AI services, an external report is one part of the evidence, not a replacement for tests on their own use cases. Confidentiality constraints and sensitive decisions still need assessment in the real operating environment.

Conclusion

Opening access to METR may improve the evidence available. Its importance will be measured in published analysis, disclosed limitations and subsequent action, not the partnership announcement alone.