A frontier model's parameters run to hundreds of gigabytes and need memory bandwidth measured in terabytes per second. Consumer hardware has neither, so the client sends an HTTPS request instead of running anything.
Made by Board & in-rack power.
Out over fibre to a facility with its own substation, chiller plant, and tens of megawatts of contracted power. Inside, rows of cabinets under overhead busway and hot-aisle containment.
Made by Site & facility systems, Board & in-rack power, Hyperscale owner-operators, Colocation & wholesale, Thermal management.
One rack, one node, one board. The package holds a compute die and HBM stacks bonded to a silicon interposer, under a cold plate that carries off roughly a kilowatt.
Made by Reference systems, Thermal management, Board & in-rack power, TSMC CoWoS advanced packaging, High-bandwidth memory (HBM), ABF substrate makers, TSMC leading-edge foundry, EUV & DUV lithography, Silicon wafers, Photoresists & wet chemicals, Masks & reticles.
The prompt is tokenised, embedded, and pushed through every transformer block. The output head produces logits over the vocabulary, one token is sampled, and the pass repeats with that token appended.
Illustrative proportions, representative of an open-weight model of this class. Not any specific released model, and not drawn to any physical scale.
One learned vector per vocabulary entry. Looking up a token is a row lookup, nothing more.
embeddingThree learned projections of the same vector. Roughly: what am I looking for, what do I offer, and what do I pass on.
query key valueOne more projection, then the result is added back into the stream. The block has done its reading.
Projects each vector up to several times its width. Most of a model’s parameters live in this step and its partner below.
feed forwardProjects back down to the stream’s width and adds the result in. The block is finished.
feed forwardProjects the final vector back out to one number per vocabulary entry. Often the same weights as the embedding table, used in reverse.
logitsEach token is now a list of a few thousand numbers. From here the model never sees words again.
embeddingOrder is added to the vectors. Without this the model would see a bag of tokens rather than a sentence.
positional encodingRescales the stream before the block reads it. Keeps the numbers in a range the rest of the arithmetic can handle.
layer normEvery position scored against every earlier position. High scores mean this token is reading heavily from that one.
attention mechanismThe values are mixed in proportion to those scores. Information moves sideways between positions here, and nowhere else in the block.
attention mechanismRescales again before the second half of the block.
layer normA simple elementwise function. Without it the whole stack would collapse into a single matrix multiply.
feed forwardOne last rescale after every block has had its turn.
layer normOne raw score for every token the model knows. Around 128,000 numbers, most of them very low.
logitsTurns those scores into probabilities that sum to one. Now the model has an opinion about what comes next.
softmaxSubmitted text is tokenized for real. The continuation is a scripted walkthrough, not a live model. The tokenizer is a byte-pair encoder of the kind these models use, not the exact one behind any particular assistant.
Scripted walkthrough. The token split is real, produced by a byte-pair tokenizer. The probabilities and attention weights are hand-authored to be representative, not measured from a running model.
The prompt is cut into 5 pieces from a fixed vocabulary of 128,000. Common words survive whole, rarer ones break apart.