Documents · Google
Pinned the model version after a silent update broke invoices
● Worked
What happened
We extract fields from supplier invoices. One Tuesday the extraction started putting the VAT amount into the total. We had not changed anything. The provider had updated the model behind the name we were calling. Our accounts team caught it two days later, after 300 invoices.
What it cost
Two days of manual corrections for two people. The API bill did not change at all.
What we fixed
We now call a pinned model version, keep 200 real invoices as a frozen test set, and run it before every model change, including the provider’s.
The lessonPin the model version and keep a frozen test set.
Reality numbers
- Team
- 11–50 people
- Monthly AI spend
- €100–1k
- Still running after 90 days
- yes