A lot of engineering leaders are walking into the trap.
They roll out AI coding tools. Engineers report that some work feels faster. Pull requests show up sooner. Someone asks for the ROI story, and the easiest answer is velocity.
That answer is risky because velocity gives leaders visible evidence: tickets closed, code generated, work in motion. But it mostly shows where motion happened. It doesn’t show whether value moved.
AI can accelerate a local activity while the delivery system stalls. Implementation finishes faster, but review queues grow, QA gets overloaded, product feedback arrives too late, and customers keep waiting.
The team feels busier. Activity rises. The business outcome barely moves.
That’s the measurement problem. If the dashboard only captures activity, it can make the AI rollout look successful while the delivery system is still constrained.
Why measurement choices become management choices
Productivity metrics carry weight. Goodhart’s Law applies quickly here: once a measure becomes the target, people optimize for the measure, and the organization stops learning from what the metric was meant to reveal.
That concern shows up often in developer productivity conversations: are metrics helping leaders understand the system, or pressuring the people inside it?
That’s the first decision. Before choosing the dashboard, decide what measurement is for.
A useful measurement system should help answer better questions:
- Where is work accumulating?
- Which handoff is slowing down value?
- What improved because of AI?
- What got worse because of AI?
- What changed for customers?
- What are we now able to do that we couldn’t do before?
If measurement can’t answer those questions, it’s probably a reporting system with nicer charts.
AI value has more than one dimension
Velocity gets airtime because it’s simple to ask about, even when it’s hard to prove cleanly. Did the task take less time? Did the engineer complete more tickets? Did the team ship more?
Those signals are useful, but incomplete.
For AI in engineering, I’d measure at least four dimensions:
- Velocity: same scope or comparable work in less elapsed time.
- Capacity: additional valuable work delivered without adding headcount or increasing unsustainable load.
- Capability: work an engineer or team can now take on that was previously blocked by skill, context, cost, or time.
- Quality: delivered work holds up with fewer defects, less rework, and stronger standards.
Velocity is the cleanest speed claim: comparable scope, less elapsed time. AI helps with boilerplate, refactoring, test scaffolding, documentation, or a narrow implementation path. That matters. But velocity should be measured across the delivery system, not only at the coding step. If build time drops and QA time doubles, the system didn’t get faster. The bottleneck moved.
Capacity is additional throughput without adding people. It may come from engineers moving faster, autonomous agents handling bounded work, internal tools reducing coordination load, or AI workflows taking on repeatable tasks. The question becomes: can the system deliver more valuable work without pushing hidden load onto the same people?
Capability has an individual and a system dimension. At the individual level, AI helps an engineer cross a boundary that used to block them: an unfamiliar language, a domain they only partly understand, or maintenance work that previously required an SME. That can be useful, but the specialist doesn’t disappear. For unfamiliar or high-risk work, the SME should still be in the review path.
At the system level, capability means the organization can attempt work that was too slow, expensive, or impractical before. A team can prototype three approaches before choosing one. Support engineering can generate better diagnostics. Platform teams can turn more operational knowledge into reusable workflows. The point isn’t that everyone does more work. The point is that the organization has a wider range of responsible options.
Quality is the dimension many teams under-measure until the cost shows up somewhere else. Defects become support load. Plausible-but-wrong code becomes rework. Weak tests make releases slower because teams stop trusting the path to production. Customers feel it as regressions, incidents, broken workflows, or slower resolution times.
A velocity-only story can make AI look successful while quality degrades. A capacity-only story can celebrate more work in flight while ignoring who or what absorbs the extra work that clearing bottlenecks creates. A capability story without review and quality control can turn unfamiliar work into hidden risk.
A complete ROI story needs all four.
And the four dimensions should tie back to business outcomes: time to market, customer lead time, incident recovery, support volume, churn risk, CSAT, NPS, revenue enablement, or whatever outcome the AI workflow is supposed to improve.
I’d also look at flow distribution. Is AI helping more time move toward value creation? Or is it increasing risk work, rework, quality and cleanup? That distinction matters because a team can look more productive while spending more of its capacity cleaning up after speed.
AI can make engineers faster while the system gets slower
Watch for local acceleration that slows the whole system.
If development accelerates but review capacity stays fixed, the queue moves to review. If implementation accelerates but requirements stay vague, the team iterates faster around the wrong thing. If code generation accelerates but test infrastructure is weak, QA absorbs the risk. If prototyping accelerates but portfolio decisions stay slow, more ideas pile up without a path to adoption.
I learned this lesson before AI was part of the conversation. On one team, engineering throughput improved faster than QA capacity. Cards piled up at the same point in the flow until moving work through the queue became its own coordination job.
The counterintuitive move was to stop feeding the bottleneck. A simple WIP limit improved cycle time because it forced us to manage the constraint instead of celebrating more output upstream.
That’s why I prefer cycle time over activity as a signal.
Activity asks, “Did people do more?”
Cycle time asks, “Did value move through the system faster?”
Lead time, change failure rate, deployment frequency, recovery time, developer experience, satisfaction, and collaboration signals all have a place. DORA, SPACE, DX Core 4, and DevEx-style frameworks help because they resist the fantasy that productivity is one number.
But every acceleration metric needs a balancing counterpart.
If you measure speed, also measure quality. If you measure throughput, also measure WIP and load. If you measure AI adoption, also measure rework and review burden. If you measure output, also measure whether customers or internal users experienced a better outcome.
Otherwise the dashboard can reward the system for moving the mess somewhere else.
AI measurement questions I’m asking
When I’m evaluating AI impact, I start with five questions:
- What business outcome should improve if this AI workflow works?
If the answer is vague, the initiative isn’t ready to scale. “Engineers will be faster” isn’t enough. Faster at what? Shorter customer lead time? Better incident response? Faster discovery?
- Where is the current constraint?
If the bottleneck is QA, faster code generation creates more downstream pressure. If the bottleneck is product clarity, faster implementation creates faster rework. If the bottleneck is technical debt, AI may make it easier to add to the pile unless the workflow also improves cleanup.
- Which dimension of AI value are we trying to improve?
Velocity, capacity, capability, and quality require different evidence. A workflow designed to improve capability may not show up as faster cycle time immediately. A workflow designed to improve quality may reduce defects while lowering apparent throughput in the short term. If you only reward speed, you may discourage the AI use cases that improve resilience, learning, or quality.
- What would make us stop or redesign the workflow?
Every AI adoption plan needs a kill metric. If review load increases by 30%, escaped defects rise, cycle time stays flat, or engineers spend more time correcting generated code than writing it, the answer should be redesign or sunset.
- What are humans compensating for today?
AI exposes weak process faster than humans do. If engineers are quietly navigating unclear requirements, missing documentation, or product ambiguity, AI may increase the load and slow the system down.
A better AI productivity story
A stronger AI productivity story can’t stop at, “AI helped us move faster.”
It should sound more like this:
“We’re measuring whether AI improves the delivery system, rather than only whether one activity got faster. We’re watching speed, capacity, capability, and quality: what moves faster, what the system can absorb, what engineers can now take on, and whether the work holds after release.
Before scaling the workflow, we know the constraint we expect it to improve and the signals that would tell us to scale it, redesign it, or stop.”
That’s more credible because it treats AI as an operating decision, not a tool adoption campaign.
If your AI strategy can only prove that one activity got faster, you do not yet know whether the organization got better.
And if you only measure velocity, AI will teach you the wrong lesson.