Inside the hall where the model runs.

A rack of dense accelerators draws in the tens of kilowatts where a rack of ordinary servers drew a few. That one number decides the building: the size of the utility feed, how much battery sits between the grid and the floor, and the thickness of the busbar overhead. New halls are planned in megawatts, not square metres.

Near enough every watt a chip consumes leaves it as heat, so the cooling plant is sized by the power draw rather than by the floor area. Racks face each other across a cold aisle: cold air up the front, hot air out the back, and the two never allowed to mix.

At accelerator density that stops being enough and the coolant goes to the chip — cold plates on the package, or the whole board immersed. A hall that cannot reject the heat cannot use the silicon it paid for, which is why cooling, not compute, is usually the binding constraint.

An idle accelerator still costs its share of the building, the power contract and the capital. The scheduler’s job is to leave as little of it idle as possible — harder than it sounds, because work does not arrive evenly and a large job needs several cards at once. In the hall below, a job needing five GPUs waits until five are free rather than starting on three.

• Batching — several requests put through together, because the cost of a pass is dominated by moving the weights, not by the extra sequence.
• Placement — cards that talk to each other belong in the same box on the same fast link, not across the hall.
• Queueing — when everything is full, new work waits. That wait is latency the person at the other end feels.

Traffic inside a hall mostly goes sideways — between machines — rather than in and out. That is why the fabric is built as leaf and spine: every rack reaches every other rack in the same small number of hops, so where a job lands matters less than it otherwise would.

Because all four end up as one number: how long the answer takes. A planner with a question is waiting on a network hop, a queue, a batch and a forward pass — and the queue is usually the biggest piece of it, not the model.

Which is why the honest answer about where a model should run is it depends on the decision. A gate decision has to survive the network being down, so it is made locally against the site’s own record. A planning question can afford the round trip. Putting both in the same place is the mistake — and the one that will not be made is the one where a barrier waits on a data centre.