Ops automation · OpenAI, Anthropic
A retry loop spent a month of budget over one weekend
○ Didn't work
What happened
A nightly job classified incoming documents. On Friday a malformed file made the model return an answer our parser rejected, and the job retried without a limit. By Monday it had made about 400,000 calls on the same file. Nothing alerted, because the only budget check was the monthly invoice.
What it cost
Roughly €9k over the weekend, about one month of our normal AI spend.
What we fixed
A cap on retries per item, a dead-letter folder for files that fail three times, and a per-key daily budget alert that pages whoever is on call.
The lessonEvery loop gets a cap, and every key gets a daily alert.
Reality numbers
- Team
- 11–50 people
- Monthly AI spend
- €1k–10k
- Still running after 90 days
- yes