Where is my server? A dashboard should never guess

Here is a class of bug worth naming, because it is easy to write and hard to see: a system stores a record of what was requested, then presents it as a record of what exists. Nothing crashes. Every page renders. The number is just wrong, quietly, for as long as nobody checks it against reality.

We had one. It concerned the region a server runs in, which — as we argued separately — is precisely the kind of fact a customer may one day need to state in writing. This is what happened and what we changed.

The bug

Our platform lets an operator set a default region for a cloud provider. Reasonable feature. The code that called the provider's API read it like this, in pseudocode:

region = configuredDefault or whateverTheCustomerChose

So when a default was configured, it won — and the customer's choice was discarded at the last possible moment, inside the function that talks to the provider. The server really was created in the default region. That part worked exactly as written.

The problem is what was not written. Nothing recorded the substitution. The order row still carried the region the customer picked, the server row was created from the order row, and every screen read the server row. So:

• the dashboard named the wrong region,
• the invoice line and the receipt named the wrong region,
• the manifest used to rebuild a server named the wrong region,
• and block storage — which reads that same field to decide where to create a disk — created the disk in a region the server was not in.

That last one is how it surfaced. A volume can only attach to a server in its own region, so the provider refused the attach with a 422. What the customer saw was "could not add storage", with no indication of why, on a server whose displayed region looked perfectly correct.

The part that stings

The provider had told us the answer the whole time. Its create-droplet response includes the region the machine was actually placed in. We stored that response — the field was sitting in the database, in a JSON column, from the moment the server came up.

Nothing read it. Every consumer preferred the foreign key that came from the order, because the foreign key is the thing you join on and the JSON blob is the thing you forget is there.

What we changed

Four rules, now enforced in one place instead of assumed in twelve.

1. A default is a default, not an override

The operator's default region now does what the word means: it preselects the region on the order form, and supplies one to resources created with no server to inherit from. It never replaces an explicit choice. Which regions a customer may choose from is a separate, explicit setting — a region is either offered or it is not.

A silent override is worse than a validation error in every case. If a choice is not permitted, the honest answer is to refuse it and say so; the one thing a system must not do is accept the input, do something else, and report back the input.

2. Ask the thing itself

After a server is created, we read the region out of the provider's own response and correct the stored record to match. Same again after the polling step, because some providers only report final placement once the machine is running.

In the normal case this changes nothing — the two already agree. Its value is entirely in the abnormal case, which is exactly when a dashboard is least able to afford being confidently wrong.

3. Dependent resources follow the machine, not the paperwork

Anything regional that attaches to a server — a block volume, a load balancer — now derives its region from where the server is, not from the order that created it. Where the two ever disagree, physical reality wins, because it is the only one the provider's API agrees with.

4. Impossible combinations get a sentence, not an error code

Attaching a volume in Frankfurt to a server in New York is not a transient failure to be retried. It cannot work. We now record which region every volume lives in and check it before calling the provider, so you get a sentence naming both regions instead of a number from someone else's API.

The generalisable bit

If you build on cloud APIs, three things travel beyond our codebase:

• Store the API's answer, not your request. They are different records and only one of them describes the world. Keep both if it is useful — but be deliberate about which one you show a human.
• Reconcile after the fact. A create call is a proposal. Read back what happened and correct your record. It is a few lines, it usually does nothing, and the day it does something it saves you an incident.
• Never let a default silently outrank an explicit input. If you catch yourself writing configuredDefault or userChoice, you have written an override and named it a default, and someone downstream is about to believe the wrong thing.

Why we are writing this down

Partly because the engineering is generally useful. Mostly because absorbing this category of problem is what a managed platform is for. If you run your own servers, you carry every discrepancy between what you asked a cloud API for and what it did — and you find them the way we did, from a downstream symptom that looks like something else entirely.

Paying someone to run the layer above your server should mean those get found and fixed once, on everyone's behalf, by people who read provider API responses for a living. When we get one wrong, saying so is part of the same deal.

Keep reading

How to choose where your server runs Latency, data residency, and what else has to share the region. Why your price does not change with the region The reasoning, and the one thing that would change it. Managed hosting vs self-managed VPS What you are really buying when you pay for management. What is included Everything above the machine, at every server size.