Attention Is All You Need: the paper that shaped modern AI

A good share of the current wave of artificial intelligence, from coding assistants to task-running agents, has its roots in a 2017 scientific paper. "Attention Is All You Need", published by eight Google researchers, introduced a neural network architecture called the Transformer and became one of the most influential works in the recent history of computing. At Espresso Labs we follow this research closely because it is what lets us separate real progress from marketing when we design AI solutions for clients. This post explains why the Transformer was a milestone and organizes, in a glossary with the original references, the concepts that grew out of it.

The world before the Transformer

Until 2017, the best language models were recurrent networks (RNNs and LSTMs), which processed text word by word, in sequence. That created two structural problems: training was hard to parallelize, because each step depended on the previous one, and the model forgot distant information, struggling to connect the beginning and the end of a long text. The attention mechanism already existed as a patch for that second problem. What the 2017 paper did was throw away recurrence and keep only attention, hence the title: attention is all you need.

What the paper changed, in practice

The Transformer architecture brought three consequences that explain much of what came next:

1. Full parallelization. Without the sequential dependency, training could be distributed across thousands of GPUs at once. That is what made it economically viable to train models with billions and later trillions of parameters.
2. Real context. Self-attention lets each word "look" at all the others at once, capturing long-range relationships that recurrent networks missed.
3. One architecture for many things. The same design that worked for translation proved effective for text, code, images, audio and proteins. AI research, previously fragmented by domain, largely converged on a single foundation.

The timeline connecting the paper to the present: the Transformer (2017) opened the way for BERT and the first GPT (2018), which led to GPT-3 and in-context learning (2020), then to ChatGPT and human-feedback alignment (2022), and from there to today's reasoning models and agents. A good part of what exists today rests on that foundation.

A glossary of modern AI, with the original references

The terms below form the vocabulary of anyone working with AI today. Each comes with the research that established it, because the popular meaning of a term often drifts away from what the science demonstrated.

Attention mechanism

A technique that lets the model dynamically weigh which parts of the input matter for each part of the output. Introduced for machine translation by Bahdanau, Cho and Bengio (2014) and elevated to the central component of the architecture in the Transformer by Vaswani et al. (2017), in the form of self-attention, where the sequence attends to itself.

Embeddings

Numerical representations of words or text passages in which geometric proximity captures similarity of meaning. The seminal work is word2vec, by Mikolov et al. (2013). They are the basis of semantic search systems and RAG.

Pre-training and fine-tuning

The paradigm of training a large model on massive volumes of generic text (pre-training) and then specializing it for a task (fine-tuning). Established in parallel by the first GPT, by Radford et al. (2018), and by BERT, by Devlin et al. (2018).

Scaling laws

The discovery that model performance improves predictably as parameters, data and compute grow, formalized by Kaplan et al. (2020) and refined by the Chinchilla work of Hoffmann et al. (2022), which showed most models were being trained on too little data. It is the economic reason behind the GPU race.

In-context learning

A model's ability to learn a new task from examples in the prompt alone, without retraining. Demonstrated at scale by GPT-3, by Brown et al. (2020). It is the foundation of the discipline of prompt engineering.

RLHF (reinforcement learning from human feedback)

A technique for aligning model behavior with human preferences, making it useful and safe as an assistant. Proposed by Christiano et al. (2017) and applied to language models in InstructGPT, by Ouyang et al. (2022), the work that paved the way for ChatGPT.

Chain-of-thought

The finding that asking the model to reason step by step before answering improves performance on complex problems, documented by Wei et al. (2022). It is the embryo of today's reasoning models, which learn to do this through reinforcement, as openly demonstrated by DeepSeek-R1, by DeepSeek-AI (2025).

RAG (retrieval-augmented generation)

An architecture that combines the model with a retrieval step over external knowledge bases, reducing hallucinations and enabling answers about private, current data. Proposed by Lewis et al. (2020). It is the standard behind most of the corporate AI applications we build.

Mixture of Experts (MoE)

A technique in which the model is divided into "experts" and only some are activated per token, multiplying capacity without multiplying the cost of each response. Introduced at scale by Shazeer et al. (2017) and simplified in the Switch Transformer, by Fedus et al. (2021). It underpins a good share of today's frontier models.

Vision Transformer (ViT)

The demonstration that the same attention architecture works for images, treating image patches as if they were words, by Dosovitskiy et al. (2020). It was the Transformer's gateway into multimodality, including the computer vision systems used in animal production that we mention in our agribusiness projects.

Why understanding AI in depth matters

Using AI and understanding AI are different things. Those who only use it depend on vendors' marketing vocabulary. Those who read the research know what each technique actually guarantees, where it tends to fail and when it makes sense for a specific business problem. At Espresso Labs, that knowledge shows up in concrete project decisions: when we mapped the new categories of work AI brought to software houses, when we choose between RAG and fine-tuning in a client's solution, and when we built Devint to evaluate developer performance in a market transformed by this same line of research. If your company wants to apply AI with that level of depth, talk to us.