A method for measuring AI engineering productivity
Token counts measure consumption. Ship Math traces a single agent or MCP call all the way to recovered engineering dollars, through five steps. Here is the full method, depicted.
The problem
Most teams measuring AI productivity count tokens. Tokens per developer, tokens per repo, growth month over month. The chart climbs and the budget review goes smoothly.
Token volume measures consumption. It carries no information about contribution. A model that retries five times, pulls in irrelevant context, and regenerates a file it already wrote burns more tokens than a model that gets the answer once. The wasteful run scores higher. The metric pays out for waste.
The chart climbs. The shipping does not. Finance asks the obvious question, and token counts cannot answer it.
The method
Each step adds one link in a chain that ends on recovered engineering cost. A call advances a ticket. The ticket carries a size. The size maps to hours. The hours convert to dollars. Ship Math measures every link, and it names the calls that earned the result.
Each step stands on its own. Tag and Bind alone beat a token chart. The full chain answers finance.
Instrument every agent call and every MCP call, and tag each with what it worked on: the developer, the repository, the tool it hit, the task it belonged to.
This is the raw event stream. It resembles a token log, with one difference that decides everything downstream. Every event already knows its job. A spec retrieval against the billing service is a distinct event from a generation against the auth service, and Ship Math keeps them apart from the first call.
For this coding session, instrument every action you take. Emit one
JSON line per agent action and per MCP call to ship-math/calls.jsonl
as it happens, with:
{ ts, developer, repo, service, tool, call_type, ticket_id }
call_type is one of: spec.retrieve | dep.trace | code.generate |
test.run | file.read. Do not summarize or batch. One event per call.
Bind the tagged calls to the ticket they advanced, then attach the ticket's story-point size to the cluster. When the ticket closes, its points flow back onto every call.
Each call gains two properties it lacked a moment ago. It has an outcome address, the ticket it served, and a size, the points the ticket carried. Productivity stops being a usage number. It becomes shipped work with a measured weight behind it.
Read ship-math/calls.jsonl. For each call, use CoreStory to
resolve the code artifacts it touched (specs, files, dependencies),
grounded in the verified spec for each artifact.
Group calls by ticket_id and attach each ticket's story points.
Return one row per ticket:
{ ticket_id, points, call_count, calls[], artifacts_touched[] }
Flag any call with no ticket_id as unbound.
Story points size the work. They do not say how long it takes by hand. Baseline supplies the missing number. Ask one or more models a direct question for the closed ticket: how long does this take a competent engineer with no agent assistance?
Run it across several models and take consensus. A single guess is soft and easy to challenge in a budget review. Agreement across independent models, scored for confidence, produces a baseline finance accepts. Subtract the assisted time and you have hours saved.
You are estimating manual effort, not doing the work. Read ticket ENG-4821: its description, final diff, and touched files. Estimate the hours a competent engineer needs to complete it with no AI assistance. Return only: { hours, confidence_0_1, reasoning_one_line } Estimate honestly. Do not pad. Run the same prompt on each model, then take the consensus and report the spread.
Hours saved per ticket already beats any token chart. Credit answers the question that makes the budget actionable: which calls earned those hours? It interrogates the call-to-ticket graph and separates the calls that moved the ticket from the ones that rode along. This is the credit-assignment problem in plain clothes. Of all the actions taken, which produced the reward?
Tokenmaxxing scores all 38 cells the same. Credit scores them by contribution: 7 calls carried the whole ticket, a 5.4× yield gap. Run it across a quarter and the pattern names which call types to fund and which to cut.
The movers share a trait. The calls that close tickets are the ones that retrieve verified context: what the legacy module actually does, which caller is hidden, where the regression hides. Generation only works when the context is already correct.
Read the call cluster for ENG-4821 (ship-math/calls.jsonl filtered to this ticket) and the final diff. Decide which calls causally moved the ticket toward close, and which were retries, dead ends, or re-reads. Return: { moved[], rode_along[], rationale_per_moved } Be strict: a call moved the ticket only if its output appears in the final work or unblocked a later step. Then summarize the mover call_types so the pattern is fundable.
Convert everything above into ratios, then into money. Points per call. Tickets per million tokens, the direct counter to tokenmaxxing. Hours saved per call.
Hours convert to dollars through one number every org tracks: loaded engineering cost per hour. Multiply, and the ledger closes on a figure. Divide recovered dollars by spend and you have the only ratio finance asks for. Dollars returned per dollar spent. Call it ratiomaxxing if you want. The ratios are the whole point.
Given for ticket ENG-4821:
points=8 consensus_baseline_h=24 assisted_h=6
total_calls=38 credited_calls=7
loaded_cost=120 tokens=1_400_000 token_price_per_m=12
Compute and return one table:
hours_saved · dollars_recovered · token_cost · roi
points_per_call · points_per_credited_call
hours_saved_per_credited_call · tickets_per_million_tokens
yield_gap
Then roll the same math up across every ticket in the quarter.
Worked example · ENG-4821
A senior engineer ports a COBOL premium-calculation routine to Java. The tooling fires 38 calls, roughly 1.4 million tokens. A token log records one number. Ship Math records the chain.
Want to move the inputs yourself? Open the interactive calculator →
The calls that move tickets are the ones retrieving verified context, not raw generation volume. CoreStory is that context layer, and it is the reason agent task resolution improves by 44% with the right context in place. Ship Math measures the difference in the currency a CFO recognizes.
Tag → Bind → Baseline → Credit → Bank
Talk to an expert → Try the calculatorFigures are illustrative and default to a representative ticket (ENG‑4821). The return-on-spend multiple divides recovered dollars by token cost alone, which reads high. Add platform and license costs to the denominator for a full picture. The ranking it produces across tickets and call types holds either way.