A dashboard says AI activity is up. That could mean more people found something useful. It could also mean they are retrying a frustrating task. The chart cannot settle the question until you know what its numbers represent.
This week supplied several unusually useful definitions. Anthropic described a way to estimate AI’s contribution to research. GitHub added feature engagement to a Copilot dashboard. Google introduced monitoring designed to identify unusual agent behavior.
Each measures a different part of the work. Putting them side by side helps explain why one impressive number rarely tells you whether a deployment is going well.
01 / THE CONTRIBUTION
Read below the headline percentage.
Anthropic reported that AI led 26% of its research and development work in August, using an internal measurement method. The method relies on a task structure, evidence about the work, model judgments, and weighting intended to reflect human effort. The company did not describe that share as fully autonomous research. [1]
The distinction matters because “AI-led” can sound like a person stepped away. A measure of contribution is not automatically a measure of independence, reliability, or the number of people a task requires. Those are separate questions.
If you want a similar measure for a small team, begin with a few finished tasks. Describe what the system contributed and what people still had to decide, check, and repair. A plain account of the work may teach you more than an early attempt to reproduce a sophisticated percentage.
02 / THE ACTIVITY
Engagement tells you where to investigate.
GitHub’s new Copilot feature-engagement view defines an active feature user as someone who used it on at least two days in a rolling 28-day period. That is a specific adoption threshold. It does not say whether the resulting work was accepted or useful. [2]
Imagine a team where use rises while review time also rises. That is a hypothetical example, but it shows why the dashboard is a starting point. You would want to look at a few representative changes and speak with the people reviewing them.
The same principle works when adoption is low. People may lack access, training, a suitable task, or confidence in the results. A low count alone cannot tell you which intervention to choose.
Give the metric a follow-up question. For engagement, ask what people attempted and whether they would choose the tool again for that task. For contribution, ask which parts of the work remained difficult. For quality, inspect the result against a standard defined before seeing it.
03 / THE RESPONSE
Monitoring needs a next action.
Google introduced agent anomaly detection in private preview. It examines logs and traces asynchronously; that timing matters because detecting a problem is not the same as preventing its first occurrence. [3]
A flagged run creates a new piece of work. Someone has to decide whether it is a real problem, contain the consequences if needed, and improve the workflow. A warning count without an owner can become another dashboard that looks active while nothing changes.
For a modest pilot, decide in advance what finding would cause you to pause. Repeated unsupported claims might trigger a source review. An action outside the assigned scope might require disabling that action until you understand what happened.
The three announcements therefore point toward a small set of connected questions: are people using the system, what is it contributing, and what happens when its work needs attention? Keep the definitions visible so a movement in one number does not quietly stand in for all three.
04 / PUT IT INTO PRACTICE
Audit one number before adding another.
Choose a metric already used to discuss an AI trial. Write its definition in ordinary language and identify the decision it is meant to support. Then inspect a few underlying cases.
If the cases cannot answer the decision question, change the measure or add one small quality check. Avoid building a larger dashboard until the first number earns its place.
A metric with a decision attached.
Build: A metric definition card, a sample-review sheet, and a decision rule.
Look for: The sampled work supports a specific decision, or reveals exactly how the measure needs to change.
The useful dashboard is the one that helps you notice something and decide what to do next.