Question Google gives site admin heart stress tests

Opened up my email this morning to get an alert from Google that we were over our 100% budget warning threshold. Now, to give you a bit of background on how this work... the Google developers platform keeps a running tally of queries, CPU seconds, data transfer totals in Mbs, and so on. Each of these elements comes at a per usage charge, tallied against how powerful our VM server is (number of CPUs, whether its SSD or hard drive, etc). All those add up to an estimated costs which, at the end of the month, is billed directly to my credit card. Based on our normal operations, I set up warning thresholds for 50% at $75 and 100% at $150. Generally, we hit the 50% warning late in the month and only rarely hit the 100% warning... especially now that I've moved us down to a less powerful server (we never crossed the 50% point on the resources for the server I set us up on originally - after all, we're only running a web site, not serving business applications or game development).

Anyway, for us to hit the 100% mark this early in the month meant a large unexpected charge. And, indeed, there was a $130 charge for Cloud DNS queries among the usual numbers. Our usual charge for Cloud DNS queries / zone management is... about 4 cents per month. For us to get a charge of $130, we'd have to have somewhere in excess of 50 million queries, given that we only have four cloud zones (an Enterprise server with 200 zones and 10 million queries pays about $43 / month. And we got charged $130... since yesterday.

Needless to say... I turned off Cloud DNS zone management. And sent Google a WTF?!!?!!?!?!! letter.

It shouldn't impact anything on the user end of things. It was only on for a VM migration convenience...

I turns out that there's a known issue affecting a number of accounts... reporting a false value/charge for queries. It's supposed to be a high priority fix that they are working on and will update people over the next couple days. So with luck, it won't even be a real charge...and my charge card can rest easy.

Like most of my avatars, I dug it up at alphacoders in the Avatar Abyss.

Nope, they had it all worked out in about 48 hours and it was all good. Very positive customer service experience too. They sent me regular emails with updates on what they were doing to solve things, with the option to pull my support ticket out of the mass fix if it didn't sound like the issue they were working on really matched up with the one I was facing. I was impressed.

Oh... here's an example. This is an email I got today as a follow up.

We are writing to let you know the results of our analysis of the issue which occurred with Google Cloud DNS billing around the 13th and 14th June. The issue was an error in the costing calculations for Cloud DNS, which caused customers' Google Cloud billing accounts to register excessive costs for their use of the Google Cloud DNS service. We sincerely apologize to all our customers who were affected by this. We have corrected the billing records on all affected accounts, and we have provided full compensation in the minority of cases where the error resulted in a financial charge to the customer.

The root cause of this issue lay in work being undertaken by the Google Cloud Platform engineering team to increase the flexibility of the Cloud Platform billing system. At present, Cloud Platform usage charges are calculated and applied to customers' billing accounts once per day. In future, charges will be calculated on a faster cycle, giving customers much more fine-grained control of their Cloud Platform resource usage and costing. As part of this work, many of the time units used in internal billing calculations are being changed, in many cases from days to seconds.

In the case of Cloud DNS, an incidental change introduced a discrepancy between the time units being used by the Cloud DNS and the billing system. Cloud DNS reported customers' managed-zone usage in seconds, but the billing system interpreted this figure as a metric in days, causing the calculated charge to be inflated by a factor of 86400 (the number of seconds in a day).

If you have any questions about your case, you can reply to this message and a support agent will follow up with you. If you no longer think this is your issue, please click the link below so that we can continue working with you to identify your issue:

*link cut so no one accidentally restarts the help case*

As I said before... impressed.

yep i can just imagine the eyebrows on some of the busier sites. Figure our little site racked up about $100 more than expected overnight before it hit my alert notification level. Imagine how high those numbers got on some of the really big site accounts.