More and more teams are pointing an LLM at their raw delivery data, straight out of Jira or Azure DevOps, and asking it for their flow metrics. Cycle time, throughput, a forecast for the quarter. The answer comes back in seconds, formatted and confident, and it usually looks completely plausible.
I understand the appeal. Chat is quickly becoming the way people want to work with their data, and the teams doing this are asking exactly the right questions. The problem sits one step earlier, in a place most people never look: whether AI flow metrics calculated from raw data can be trusted at all.
Why an AI Metric Can Look Right and Be Wrong
Take the simplest flow metric there is: cycle time. On paper it’s a subtraction. The day work finished minus the day it started.
In real delivery data, that subtraction hides a long list of decisions, and every one of them moves the result.
Which workflow states count as “started” and which count as “finished”? Every team maps these differently, and getting the mapping wrong shifts every measurement in the dataset. What happens when an item moves backward, from review into development again? If the clock resets, rework disappears from view. What about an item that jumped from the backlog straight to done? Counting it as zero days drags the whole metric down, and it may simply be a gap in the data. A ticket that was closed and reopened two weeks later: one item or two? Does time spent waiting count, or only the time someone actively worked on the item? On most boards, waiting is the majority of the total, so this single choice changes the number more than any other.
And then the last one: does the AI report the average or a percentile? Most items finish fast and a few drag on for weeks, so the handful of slow ones pull the average up. The number you can safely commit to is the percentile, the one that 85 of your last 100 items came in under. An average and an 85th percentile from the same data can be days apart.
An LLM answering from raw data makes every one of these decisions on its own, out of sight, and reports none of them. The number arrives clean either way, so a right answer and a wrong one are indistinguishable on your screen. Ask again tomorrow and the answer can change, because nothing forces the model to make the same decisions twice.
The Engine Between Raw Data and a Number You Can Trust
Here’s the comparison I keep coming back to. Hand an AI ten years of your raw bank transactions and ask it for an audited financial statement. You will absolutely get one. Every column will add up, the formatting will be flawless, and you will have no way to tell whether any figure on it is true. Nobody would take that document to an audit. Yet a cycle time chart produced the same way goes into leadership decks every week.
The engineers in your organization will see the fix quickly, and they’re right about the direction. Check a few numbers by hand. Fix the definitions in the prompt so the model can’t improvise. Better yet, have it write real code for the calculation. Each step is reasonable. And each step, followed to its end, arrives at the same destination: a fixed set of tested rules that every question must pass through, applied the same way on every run. That has a name. It’s a data engine.
Building one is a legitimate choice, and some teams should make it. Just be clear about the size of the decision. Wiring an LLM to your tracker’s API is days of work. Knowing which edge cases exist, which ones matter, and how each should be resolved, that’s what takes years, and it’s the part that never shows up in the demo.
Run a Definition Audit on Your Own Metrics
You can test all of this on whatever produces your delivery numbers today, whether that’s a tool, a spreadsheet, or an AI experiment someone on your team is already running. It takes one afternoon.
Pick one team’s board and ask your source six questions. What counts as started, and what counts as finished? What happens when an item moves backward? When it skips states? When it’s reopened? Does waiting time count, or only active work? And is the reported number an average or a percentile?
A source you can trust answers all six, the same way every time, and can show you how it got there. If yours can’t, you now know precisely what needs to be settled before its numbers go anywhere near a commitment.
That layer of settled, tested definitions is exactly what we’ve spent the past eight years building at Nave. Every metric runs through the same validated engine on every query, and our MCP server lets you point your own AI tools at those metrics instead of at raw data, so the chat you want comes with numbers you can defend. You can see it at getnave.com.
Your AI will always give you an answer. Before you plan around it, make sure something you trust did the math.
I’ll see you next week, same time and place for more managerial goodness. Bye for now!
