Cost per token: the number nobody watches until it hurts

An internal assistant went from 400 dollars to 6000 dollars a month in six weeks without a single code change. Nobody had changed the model, the prompt, or the feature set. Something else was happening.

Usage grew, and with it, the context resent on every turn grew. A conversation after three days of history meant resending everything each call. Nobody had set a context limit or a conversation retention policy.

The cheap thing to do would have been to cap history. But the team wanted the full conversation available. So the bill climbed.

Instrumenting cost per conversation rather than per month made the problem visible in a day. A monthly bill is an abstraction. A conversation that costs 12 dollars is real. Suddenly the issue was visible and urgent.

Trimming history and caching the system prompt put the bill back where it belonged, with no change in perceived quality. By reducing the redundant tokens resent on every call, we cut costs in half without removing features.

The lesson: instrument what you mean to control. If you measure cost per month you'll miss the conversations that spiral. If you measure per conversation you catch outliers immediately.