What is the transformer architecture? Basics, variants, market status May 2026

What is the transformer architecture?

The transformer architecture is a neural-network design for sequence processing introduced in 2017 in the paper "Attention is all you need" by Vaswani et al. (Google Brain and Google Research). As of May 2026 it is the technical foundation of every relevant language model – GPT, Claude, Gemini, Llama, Mistral, DeepSeek, Qwen – and also the basis of many image and audio models.

The central building block is the attention mechanism. Instead of processing word-by-word sequentially like older RNNs and LSTMs, a transformer looks at all positions of a sequence in parallel and decides via self-attention which positions relate to which. This has two practical consequences: computation is parallelisable (which made massive GPU training feasible in the first place), and the model captures long-range dependencies better than the old recurrent architectures.

The transformer was originally developed for machine translation – a classical encoder-decoder setup. Since then the field has settled on three main variants. Encoder-only models (BERT, RoBERTa, DeBERTa) understand inputs and produce embeddings or classifications. Decoder-only models (GPT family, Claude, Llama, Mistral) generate text token-by-token and as of May 2026 are the dominant variant for chat and agent applications. Encoder-decoder models (T5, BART, mT5) remain relevant for translation, summarisation and structured sequence-to-sequence tasks.

For an SME, the term "transformer" is less an architectural choice than background information. The practical question is not "should I use a transformer" – every modern LLM is one – but "which transformer-based model via which API".

Why it matters

The transformer architecture explains why the last eight years of LLM progress were possible – and why certain limitations still apply in May 2026.

Parallelisability. RNNs and LSTMs had to process token-by-token because every step depended on the previous one. On modern GPUs they could not fully use the available compute. Transformers compute all positions in parallel and use GPU architecture efficiently. Training a hundred-billion-parameter model on a GPU cluster takes weeks instead of years. Without this efficiency jump there would be no GPT-4, no Claude, no Gemini.

Long-range dependencies. RNNs had trouble with relationships over long distances – the connection between sentence 1 and sentence 50 was lost ("vanishing gradient"). Transformers with self-attention see all positions directly and can in principle model relationships across arbitrary distances. In practice this is limited by the context window (see was-ist-context-window) and by O(n^2) memory cost, but qualitatively it is a different world from RNN/LSTM.

Scaling law. With the transformer architecture it became visible that larger models, more data and more compute lead to predictably better quality (Kaplan et al. 2020, Chinchilla 2022). The scaling law carried the industry through the GPT-3-to-GPT-4 wave and as of May 2026 is still an effective heuristic, though with a clear slowdown in pure parameter count.

Today's architectural trends. As of May 2026 a transformer variant called Mixture of Experts (MoE) dominates. Instead of one dense network of 70-400 billion parameters active at every token, an MoE model has many "experts" – subnetworks activated depending on the token. Llama 4 (Meta, April 2025), Mistral 8x22B and 8x141B, DeepSeek V3/V4 and GPT-4-Turbo use MoE variants. Advantage: same or better quality with markedly less active compute per token. Disadvantage: memory requirements rise (all experts must be loaded), engineering complexity rises.

For an SME the MoE trend means in practice: prices for high-quality models keep falling (OpenAI cut GPT-4o by 25% in May 2026; a trend that should continue through end-2026). Self-hosting models with > 100 billion parameters remains demanding because RAM requirements are high, but cloud-API costs become increasingly SME-friendly.

Mechanics of the transformer

A transformer block has four central components. Understanding them helps when reading model datasheets and talking to technical partners.

1. Tokenisation and embeddings. Incoming text is first broken into tokens (see was-ist-token) and each token is mapped to a vector (embedding, typically 768-12288 dimensions). These vectors are the language in which the transformer computes.

2. Positional encoding. Since self-attention has no inherent notion of order, the position of every token is encoded into the vector. Vaswani et al. used sinusoidal functions of various frequencies; modern models use learned or rotary variants (RoPE – Rotary Position Embedding, standard in Llama, Mistral, DeepSeek). Without positional encoding "Anna kisses Bob" and "Bob kisses Anna" would be identical to the transformer.

3. Self-attention with multi-head. The heart. From each token vector three vectors are derived: query, key, value. For every token pair the similarity between query and key is computed – that is the "attention weight" between positions. Value vectors are mixed with these weights. Result: every token gets a new representation containing information from relevant other positions. Multi-head means this happens not once but 8-128 times in parallel with different projections. Each "head" learns a different relationship pattern – syntax, coreference, semantic proximity, order. This diversity explains the depth of understanding in modern models.

4. Feed-forward network and residual connections. After the attention layer every token embedding passes through a feed-forward network (two linear layers with an activation). Residual connections and layer normalisation keep training stable. A transformer consists of N such blocks stacked (typically 24-96), each layer learning more abstract relationships.

Decoder-only detail. In GPT-style models self-attention is masked – every token may only attend to previous positions, not future ones. That gives the autoregressive character: the model generates token-by-token, each new token built on the existing sequence.

MoE detail May 2026. In Mixture of Experts a routing layer replaces the single feed-forward network. Routing decides per token which 2 of e.g. 8 experts are activated. This saves compute per token. Llama 4 Maverick (2025) has 17B active out of 400B total parameters; Mistral 8x141B activates two experts per token. As of May 2026 MoE is the de-facto standard for new frontier models.

Efficiency tricks. FlashAttention (Dao et al. 2022, FA-2 2023, FA-3 2024) reduces the memory cost of self-attention through clever block mapping onto GPU SRAM and as of May 2026 is standard in vLLM, TGI and every serious inference stack. Sliding-window attention (Mistral) and sparse attention (Longformer, BigBird) break the O(n^2) complexity for very long contexts. These tricks explain why 1-million-token context windows are technically and economically feasible in May 2026.

When this knowledge becomes practical

An SME does not build a transformer from scratch – that is a job for research labs with hundreds of millions of dollars. Still, there are four situations where architectural understanding is concretely useful.

First: model selection. When choosing between models, architectural categories help. Decoder-only models (GPT, Claude, Llama) are the choice for chat, generation, agents. Encoder-only models (BERT-style, E5, BGE) are the choice for embeddings, semantic search, classification – see embeddings-und-vektoren. Encoder-decoder (T5-style) is a niche case in May 2026 for structured sequence-to-sequence tasks. Whoever needs embeddings does not pick GPT-4; whoever needs chat does not pick BERT.

Second: understanding cost and latency. Long-context applications cost quadratically more – O(n^2) is not only theory but visible on the API bill. Sending 200k input tokens means paying not only for tokens but for the quadratically grown computation. FlashAttention and tricks dampen this, but the basic effect remains. Practical consequence: anywhere context > 50k tokens, evaluate RAG instead of long context (see retrieval-augmented-generation).

Third: judging self-hosting. Whoever considers self-hosting Llama or Mistral should know: MoE models need RAM for ALL experts, even when only few are active per token. Llama 4 Maverick with 17B active parameters still needs 400B parameters in GPU RAM. This makes self-hosting harder, not easier, than for classical dense models. Whoever wants to self-host looks at "active params" AND "total params" – see vergleich-lokale-llm-runtimes.

Fourth: reading vendor roadmaps. When Anthropic, OpenAI or Mistral announce new architectural features (long contexts, speed-ups, new attention variants), basic understanding helps tell marketing language from real improvements. Example May 2026: "1 million token context window" is real – but recall quality drops in the upper range (see was-ist-context-window). "New MoE architecture" is real – but for SME practice delivers less than the marketing curve suggests.

Rule of thumb: Architecture knowledge is background, not daily business. You need it to vet vendor datasheets, benchmarks and technical proposals from your advisors – not to train models yourself.

When this knowledge does not help

Three situations where deep-diving into architecture burns time and attention without payoff.

First: you want to train a model from scratch. As of May 2026 that is a research project with budgets of CHF 5-500m. Even fine-tuning an existing model is the wrong choice for 95% of SME applications – RAG usually achieves better results at a fraction of the effort (see was-ist-fine-tuning-vs-rag). Anyone discussing model architecture without first answering this question is going down the wrong path.

Second: you want to "tune" an existing model through architectural changes. A fiduciary asking "should the transformer have more heads for our receipts?" is asking the wrong question. Models are what they are – the variables are prompts, RAG, data, workflow. Architecture is the vendor's choice, not the customer's.

Third: you debate architecture instead of application quality. It is tempting to lose yourself in MoE activation patterns and attention-head specialisation. But for an SME what counts is: does the model reach the quality my use case needs, at acceptable cost and latency, with sufficient compliance? Those four questions are better answered by a benchmark with your real data than by any architectural analysis.

Recommendation. Read the Vaswani paper once, watch Karpathy's "Let's build GPT" as a one-hour video – and then leave architecture alone. You have more important things to do: organise data, formulate use cases, build an eval suite.

Trade-offs

FAQ

Related topics

Sources

1. Vaswani et al. – Attention Is All You Need (arXiv:1706.03762) · 2017-06
2. Dao et al. – FlashAttention-3: Fast and Accurate Attention with Asynchrony · 2024-07
3. Meta AI – Llama 4 Model Card and Architecture Notes · 2025-04
4. Mistral AI – Mixtral 8x22B and Mixture-of-Experts Documentation · 2026-03
5. Stanford CRFM – State of Foundation Models 2026 Report · 2026-04