view
view
Skip to Main Content
whiterabbit-logo
Good Engineering Tells AI Where to Go
Blog

Good Engineering Tells AI Where to Go

Last year, somewhere between four and five times the crowd we planned for showed up to vote at The Game Awards, and you could feel the system straining under them. The Game Awards is the closest thing the games industry has to the Oscars, one night where millions of people log on to crown their favorites. For a show that runs on live voting, that is an actual problem to have, right up until the system underneath it starts to strain while everyone tries to vote at the same moment. The voting held, but barely.

When your client's biggest night of the year is also the night your infrastructure gets hit hardest, there is no second take. So we went back to them with a proposal. Let us rebuild the foundation before the next show, and build it for a crowd far bigger than the one that nearly tipped us over.

From a two-story foundation to twenty floors

Think of it like a house. Last year we had a foundation poured for a two-story home. This year we needed one that could hold a twenty-story building. Same plot of land, completely different base underneath it.

We scoped this work at around 180 hours when we first looked at it, six or seven months back. That estimate assumed a person doing most of it by hand. We finished about 90 percent of it in roughly 80 hours.

That number is the part everyone wants to talk about, and it is the part I find least interesting. Saving a hundred engineering hours looks nice in a report, but the hours were a side effect of what we were really after, which was control.

Control was the whole point

Eighty hours against a 180-hour estimate is a good number, and I am happy to have it. Like I said, the bigger reason we did it this way was control.

When you set up infrastructure by hand, you are trusting that the person doing it remembers every rule, every cost ceiling, every setting. And when something breaks at two in the morning during the show, rebuilding by hand can take hours. We wanted a setup where spinning up what we need takes ten or fifteen minutes from a button, where every change gets written down in code, and where the thing fails in a way we can read and recover from instead of just falling over.

All of that lives under a name that sounds scarier than it is: Infrastructure as Code. You stop clicking around in a cloud console and start describing your whole setup in code, so it can be rebuilt, reviewed, and tracked like any other part of the product. Fewer hands on the wheel means fewer ways to crash the car.

An architect, no DevOps team, eighty hours

Here is the part I keep coming back to.

The architect who led this, has never run a DevOps team. Normally this kind of work goes to a specialist, the sort of engineer who has lived in cloud pipelines for years and can name their price on the job market because of it. We do not have a dedicated DevOps team. But our architect did it anyway, with AI carrying the parts he had not memorized, and he got it done in 80 hours on his first actual attempt at it.

What that tells me is that the design thinking matters more than the syntax. If you understand how a system should hang together, AI can fill in the commands you do not carry in your head. The judgment is the hard part, and the judgment is still human.

So who is actually in charge?

This only works because the AI never gets to make the call on its own. We built a few guardrails into the process, and they are worth walking through, because this is where most AI experiments go sideways.

The first piece is a knowledge base. Our architect spent months researching the architecture before any code got written. Rather than keep all of that in his head, he handed it to an agent and told it: this research, these decisions, this is the blueprint. Now when anyone tries to create a new service, even through code, the agent checks it against the plan first. If the plan never called for that service, it stops them. The nice side effect is that the knowledge does not walk out the door even if our architect ever does. It lives in the system, not in one person's memory.

We also refuse to let one agent do everything. One agent guards the infrastructure, with a set of rules about what a new service is allowed to look like and when it is overkill. Another agent writes code against those rules. A third reviews that code. We split it on purpose, the same way you would never let the person writing a check also be the person who approves it. Agents are like people that way. You do not hand one of them all the keys.

And the rule I will not bend on: nothing reaches production without a human looking at it. The small, low-risk changes, the kind that cannot run up a bill or take the site down, the agents handle on their own. But the moment a change touches the live environment or costs actual money, an architect reads the code by hand and approves it before it ships. Even when an agent wrote that code, a person signs off on it.

The way I think about it is plain enough. You stay the master and AI is the tool. To build the system I just described, you have to know enough to design the guardrails in the first place. The other morning one of our engineers had handed an assignment to an agent and was sitting at his desk tossing a stress ball while it ran. I asked if he planned to do any work, and he grinned and said the work was getting done because he was the one telling it what to do. That is the whole idea. He pointed the thing in the right direction, and the machine did the typing.

The AI drafts, the human decides, and that order never flips. I tell my engineers that the AI is doing the work, but you are the one running it, because the moment you let the tool make the calls, you have stopped engineering and started gambling.

What comes next

This is the start of something bigger for us. We spent the first half of this year getting every engineer a license and getting them comfortable, and most of the team is there now. The next six months are about a program I have started calling transformation, and it begins, on purpose, with the most boring work we have.

Over the next six months we are pointing the same approach at the work nobody enjoys. Monthly site maintenance is a good example. It pays well and it is repetitive, the kind of thing you could do half asleep, which is exactly why a person should not be spending their week on it. If we let AI take the boring, repeatable parts, we free our engineers to learn, to take on harder problems, and to do the work that needs a human brain.

The people who proved this can work, is our architect and the engineers next to him on this project, are the ones who will lead it. They earned the trust. And the rest of the team listens to the folks who have actually done the thing.
If you run an agency, here is what I would take from all of this. The engineers who can design a system and hold the line on quality are about to get far more done than they used to. The teams that win with AI are the ones who stay the master and let the tool do the work.

The judgment, the design, the call on what should exist in the first place, all of that stays human, and it matters more now than it used to. The thing that kept us pointed in the right direction was knowing where we wanted to go before we ever asked AI for anything.

Schedule a call with our team