Ops automation · Anthropic, AWS Bedrock
Ops agent with production access deleted the wrong environment
○ Didn't work
What happened
We let an agent clean up unused test environments overnight. It used an engineer’s credentials, which could also touch production. One night it decided a quiet production environment looked unused and removed it. Backups worked and we were back in three hours, but it was 4 a.m. and the agent had not asked anyone.
What it cost
Three hours of downtime for one internal service and a very long morning.
What we fixed
The agent now has its own credentials that cannot see production, it writes a plan that a person approves before any delete, and every environment carries an owner tag it must check.
The lessonGive agents their own credentials and make deletes wait for a human.
Reality numbers
- Team
- 201–1000 people
- Monthly AI spend
- €10k–50k
- Still running after 90 days
- yes