Journal · 2026-04-14
What belongs in a model call inventory
A call inventory is not a log dump. It is a named list of every place an app asks a model for something, written so a product lead and an engineer can point at the same row.
When we open an assessment, the first artefact is not a chart. It is a table with one row per invocation site. If that table is sloppy, every later sentence in the report argues with itself.
An invocation site is a place in the app that can ask a model for a completion, a classification, a rewrite, or a similar result. The chat composer is one site. A background job that summarises a record on save is another. A retry that fires after a timeout is not a new site; it is a behaviour of an existing one, and it needs its own column.
Each row needs a name a product person would recognise. “POST /v1/complete” is not a name. “Composer, new message, waiting user” is a name. We also record whether a person is looking at a spinner, whether the call is allowed to fail silently, and which model is configured for that path today — not which model the team meant to configure last quarter.
Retries belong beside the site, not underneath a separate heading called ‘errors’. In more than one Klang Valley app we have read, retries were the majority of the week’s completions on a single route. Folding them into ‘usage’ without a note makes the composer look busier than any person in the building.
Fallbacks are the other column people skip. If the primary model is slow and the app asks a second model, that is still the same site with a different servant. Mixing the two into a single count hides the moment the app changed its mind.
What does not belong in the inventory: prompt text, training opinions, or a wish list of features. Those conversations are cheaper after the rows exist. If a site cannot be named, we write ‘unlocated call’ and treat that as a finding, not as a row to pad.
Teams sometimes ask us to start from the provider invoice instead. An invoice is a useful check; it is a poor inventory. It will not tell you which screen spent the money. The inventory has to come from the app, then the invoice can argue with it.
If you are preparing for an assessment, try filling the table yourself for an hour. The blank rows are the assessment. Bring those blanks to the intake call.