Diagnosing a Runaway AI Bill

Nothing Wilts is a recipe site that writes itself: a scheduled job drafts a new recipe once a day, a self-hosted image model illustrates it, and a vision model checks the photo actually looks like the dish before anything reaches a git commit. It’s a nice, small automation — until someone asks “how much is this actually costing,” and the honest answer turns out to be “more than it should, and I’m not sure why.”

The wrong way to answer that question

The instinct is to read the code and estimate: one recipe a day, a system prompt of about this many characters, a response of about that many tokens — multiply it out, get a number that sounds plausible, move on. That’s exactly what a first pass at this produced: a tidy estimate of maybe 5,000–12,000 tokens a day.

The actual number, pulled from the real spend ledger, was roughly six times that on average — and climbing.

Estimating from source code tells you what a system is supposed to do. It doesn’t tell you what it’s actually doing at 4 a.m. every four hours, unattended, for a month. For that you need the ledger, not the code.

Pulling real numbers

The pipeline routes every model call through a self-hosted LLM gateway, which — because it’s backed by a real database — keeps a row per request: timestamp, token counts, cost, which model. That’s the actual source of truth, and it’s usually one query away:

SELECT DATE("startTime") AS day, COUNT(*) AS calls,
       SUM(total_tokens) AS tokens, SUM(spend) AS spend
FROM spend_logs
WHERE api_key = '<this project's key>'
GROUP BY day ORDER BY day;

Thirty days of output turned an “it’s roughly $8 a month” impression into a much clearer picture: costs were low and flat for the first week and a half, then started spiking into $0.30–$0.90 days — and the spikes clustered at almost exactly the same six times every single day.

That regularity was the tell. Six evenly-spaced spikes a day doesn’t happen by accident; it happens when a cron job runs on a fixed schedule and does the same expensive thing every time it fires, whether or not that’s actually necessary.

What was actually running up the bill

The suspect was a retry job that checks for any recipe still missing its photo and tries again — every four hours, forever, by design, so a photo generated late doesn’t stay missing indefinitely.

The part that wasn’t by design: each retry attempt did a full draft → generate → vision-check → (if rejected) revise → regenerate cycle, up to three times, before giving up for that run. Nothing about a failed attempt was remembered between runs. A recipe whose photo needed a few tries to pass the vision check would get the exact same full-price treatment again four hours later — and again after that — with no memory that it had already failed twice.

The self-hosted image generation itself was free. The two LLM calls wrapped around every single attempt — writing the image prompt, and checking whether the result actually looked right — were not, and they were the ones firing on a loop with no backoff and no exit condition.

The fix

Not a cheaper model, not a rate limit — just giving the retry loop a memory:

• Track how many times a given recipe’s photo has failed, and when the last attempt was.
• Back off exponentially between attempts (8 hours, then 16, then 32) instead of retrying on the same fixed four-hour clock regardless of recent history.
• After a handful of failed attempts, stop automatically and flag it for a human to glance at, instead of retrying indefinitely.

None of this is exotic — it’s the same shape as any retry-with-backoff pattern. The point isn’t the pattern; it’s that the pattern was missing, and nothing in the code would have told you that without looking at what actually happened, not what was supposed to happen.

The actual lesson

“How much is this costing” is an observability question before it’s an optimization question. The estimate from reading the code was off by roughly 6x, in the same direction cron-driven systems usually fail: not doing too little, but quietly doing the expensive thing more often than anyone intended. The fix took an afternoon once the real numbers were in hand. Getting the real numbers took five minutes and one query — the hard part was remembering to ask the ledger instead of the code.