Anthropic gave six people from an outside evaluation nonprofit laptops, office space, and access to internal systems roughly comparable to its own risk-assessment teams. The point was not to help them make a nicer scorecard. It was to let them see enough of the company to reach and publish an informed judgment. [1]
That is an unusually literal version of a problem showing up in ordinary AI work. The final answer is often the easiest part to see and the least revealing part to review. A clean memo can sit on top of a bad source, a skipped check, twelve unnecessary tool calls, or an uncertainty the system quietly smoothed away.
Three developments this week make the same point at different scales. Google published a method for inspecting an agent’s path, Anthropic described what meaningful outside access can require, and Sullivan & Cromwell showed how sources and uncertainty can become part of a professional work product.
Together, they suggest a simple standard. A reviewer needs more than permission to approve the answer. The work has to arrive with enough evidence to challenge it.
01 / THE PATH
A good answer can conceal a bad run.
Google’s agent-evaluation guide opens with a failure that is easy to miss. An agent can loop, call a tool repeatedly, spend more than expected, and still return a polite answer that looks fine. Reading only the final paragraph would reward the wrong behavior. [2]
The proposed fix starts with the work a person expects to get done. Write a few realistic scenarios, keep the test set stable, and inspect the execution traces: prompts, tool choices, arguments, timing, cost, and output. Turn the failures people actually find into checks that can run again.
This does not need to begin as an elaborate evaluation program. For a recurring research brief, the first review packet might contain the draft, its source list, the searches or tools used, the items excluded, and the time or cost of the run. A second person should be able to see whether the process supports the conclusion.
That changes review from taste to diagnosis. “This looks right” becomes a starting observation. The useful questions are which source carried the claim, which step failed, and whether a correction will survive the next run.
02 / THE ACCESS
A reviewer needs enough access to disagree.
Traces and sources help only if the reviewer can reach them. Anthropic’s arrangement with the nonprofit METR makes that requirement concrete. Anthropic says the six-person embedded team can inspect much of what its own risk teams can see and publish assessments without Anthropic’s approval. Anthropic can require redactions within predefined categories, and METR can say whether it believes those redactions were justified. [1]
This is an announced oversight arrangement, not evidence that the oversight has worked. METR’s future publications, access in practice, and any redaction disputes will matter more than the launch description.
The design still exposes a useful tension for less dramatic work. A reviewer who sees only the approved summary is not independent in any meaningful sense. A reviewer who receives every customer record, private prompt, and credential creates a different problem.
Access should follow the decision. Someone checking a research brief may need the public sources, the search record, and the discarded claims. They probably do not need unrelated private files. Someone approving an automated account change may need the proposed action, permission used, before-and-after state, and rollback path.
The reviewer must be able to challenge the evidence without inheriting every secret in the system.
03 / THE WORK PRODUCT
Bring the evidence to the decision.
Sullivan & Cromwell’s Agreement Analyzer offers a more familiar place to land the idea. The law firm’s tool combines a draft agreement with selected precedents, negotiating playbooks, client positions, and market-practice context. It returns a memorandum and proposed tracked changes for a lawyer to review, revise, or reject. [3]
The account also describes a potential process in which agents challenge one another’s findings. The work product would distinguish findings that survived the challenge, findings that were dismissed, and unresolved issues kept with qualifications. A lawyer could inspect the explanation and the precedent before deciding what belongs in the final markup.
That is more useful than attaching a confidence score to a finished answer. The evidence sits next to the proposed change, where a responsible person can act on it. Uncertainty is not a warning added at the bottom. It is part of the work being reviewed.
The source is a joint vendor-customer case and provides no independent accuracy or productivity measures. It does not prove that multiple agent passes create better legal work. It gives us a work-product pattern worth testing: show the proposal, the authority behind it, what remains unresolved, and the person who owns the decision.
This also reveals the cost of serious review. Evidence has to be collected, protected, and made legible. Somebody has to investigate exceptions. A review packet that nobody opens is another report. The test is whether it helps a person catch a weak source, a missed step, or a hidden cost before the work travels farther.
What this week’s sources don’t establish
They do not prove that one evaluation stack, access model, or review format will transfer to every organization. Google published engineering guidance, Anthropic announced an arrangement whose results are still ahead, and the law-firm example has no independent performance audit. The practical standard is smaller: make the evidence available, then see whether it changes the quality of review.
04 / PUT IT INTO PRACTICE
Build the packet before you improve the prompt.
Choose one AI-assisted deliverable that another person could reasonably accept or reject. Assemble a one-page review packet with the output, the sources or records behind it, the important steps and exceptions, the uncertainty that remains, and the person responsible for the decision.
Run a small comparison. Ask one reviewer to inspect the output alone and another to use the packet. Record which problems each person finds, how long the review takes, and which evidence changes the decision. Remove anything that adds exposure without improving the review.
One deliverable. Enough evidence to challenge it.
Build: A one-page AI review packet for one real or hypothetical deliverable, with its output, sources, path, uncertainty, and decision owner.
Look for: A second person finds a weak source, missed step, unresolved issue, or hidden cost faster than with the final output alone, without receiving unrelated private data.
If the packet gives a reviewer no better question to ask than the answer did, it is decoration. Keep the evidence that can change the decision.