I keep seeing posts on LinkedIn about runaway Google Cloud costs. It's becoming a pattern. And I think I know why: AI has made it almost too easy to spin things up. A few prompts, a bit of Terraform, and suddenly you've got services running that you didn't fully think through cost-wise.
I can relate. I want to experiment, play with AI, try new things, see what's possible. But I also can't have my GCP bill quietly climbing to $500 while I'm not paying attention. So I built something to handle it.
The Problem With GCP Budgets Out of the Box
GCP has budget alerts built in. You can set thresholds at 50%, 80%, 100% and it'll email you. That's fine - but it doesn't stop anything. It just tells you the house is on fire while you're already outside watching it burn.
The other catch: actual spend data in GCP has up to a 24-hour lag. So even if you set an alert at 100% of actual spend, you could blow past your budget before the system even knows it happened.
The Google-Recommended Fix: Detach Billing
Google actually documents a pattern for this - using a Cloud Function triggered by a budget Pub/Sub notification to programmatically detach the billing account from the project. No billing account = no more charges. All services stop.
It's the nuclear option, but for personal projects or sandboxes, that's exactly what you want.
I put together a repo that implements this with Terraform: github.com/jeremydegardeyn/gcp-billing-cap
It provisions:
- A GCP budget with email alerts at 50% and 80%
- A Pub/Sub topic wired to the budget
- A Cloud Function that detaches billing when the threshold is hit
- The SMTP alert email so you know it happened
Why I Trigger on Forecast, Not Actual Spend
Because of that 24-hour lag, I don't trigger on actual spend hitting 100% - I trigger on Google's forecasted spend crossing the threshold. GCP projects your monthly spend based on current burn rate and publishes that in the same Pub/Sub payload (forecastThresholdExceeded). If the forecast says I'm going to hit my cap, I'd rather cut it off early than find out tomorrow I already blew past it.
It's in Dry Run Mode by Default
I'm not quite ready to have services yanked out from under me automatically, so the billing detachment is commented out by default. The function fires, sends the email, logs what it would have done - but doesn't actually pull the plug. When I'm confident in the setup, I'll uncomment one line and redeploy.
If you're running personal GCP projects and experimenting with AI, give it a try. It's a small setup cost for a lot of peace of mind.
0 Comments
Leave a Comment