ngosangns Knowledge Base

Transformer Architecture

#technology#transformer#attention

Updated: 2026-09-24

Source

This note summarizes the 2017 paper "Attention is All You Need" and explains how the Transformer works. It starts with the limits of RNNs, then walks through the encoder and decoder stacks, attention, and positional encoding. You get the intuition behind query, key, and value, plus the scaled dot-product formula and why multi-head attention helps. Later sections give the reasons the architecture succeeded and describe variants such as BERT, the GPT series, Transformer-XL, Longformer, efficient models, ViT, and multimodal work. The final part covers applications outside NLP, from vision and speech to time series and drug discovery. Read it top to bottom for the full picture, or jump to the attention section when you need a refresher.

Summarized from: https://bfcmath.github.io/posts/Attention-is-all-you-need-and-much-more/

Introduction

  • The 2017 paper "Attention is All You Need" is the foundation of the Transformer architecture, which revolutionized modern AI
  • The Transformer replaces RNN/LSTM/GRU models in sequence modeling, especially machine translation
  • The architecture avoids recurrence and relies entirely on the attention mechanism to establish global dependencies

The Problem With RNNs

  • RNN: Processes words one at a time, is slow on long sequences, and is hard to parallelize during training
  • Transformer: Processes all words in parallel and can "see" the whole sentence/paragraph at once, so it is faster and does not forget information

Transformer Architecture

Encoder-Decoder Architecture

  • Encoder: The "reader" - understands a sentence in one language and builds a representation
  • Decoder: The "writer" - uses the representation to generate a sentence in another language
  • Both the encoder and decoder have 6 stacked layers

Encoder Structure

Each encoder layer consists of:

  1. Multi-Head Self-Attention: Looks at all words in the sentence at once and decides which are important
  2. Residual Connection & Layer Normalization: Skip connection plus normalization
  3. Position-wise Feed-Forward Network: A feed-forward network applied at each position
  4. Residual Connection & Layer Normalization: Skip connection plus normalization

Decoder Structure

Each decoder layer consists of:

  1. Masked Multi-Head Attention: Self-attention with a mask so it cannot look at future words
  2. Residual Connection & Layer Normalization
  3. Multi-Head Attention over Encoder Output: Attention over the encoder's output
  4. Residual Connection & Layer Normalization
  5. Position-wise Feed-Forward Network
  6. Residual Connection & Layer Normalization

Key Components

  • Layer Normalization: Normalizes the output of each layer, managing internal covariate shift
  • Residual Connection: Skip connections that avoid vanishing gradients and make it easier to train deep networks
  • Masked Attention: In the decoder, masks future words so it only looks at words already generated

Attention Mechanism

Dot-Product Attention

  • Query (Q): "What am I looking for?"
  • Key (K): "What am I?"
  • Value (V): "The actual information I hold"

Formula:

Attention(Q, K, V) = softmax(QK^T / √d_k) V
  • Compute the similarity between Q and K
  • Scale by √d_k to avoid overly small gradients
  • Apply softmax to produce attention weights
  • Take the weighted sum of V

Multi-Head Attention

  • Instead of a single attention, use multiple "heads" (eyes) that look at the sentence in different ways
  • Example: one head focuses on "dog", one on "bark", one on "pet"
  • Lets the model gather fuller information from multiple perspectives

Scaled Dot-Product Attention

  • The scale factor (√d_k) helps:
    • Avoid overly small gradients when d_k is large
    • Stabilize training
    • Improve performance

Positional Encoding

  • Attention has no inherent notion of order
  • Positional encodings are added to the embeddings so the model knows the position of each word
  • Uses sin/cos functions with a different frequency for each dimension

Why Did the Transformer Succeed?

Common Reasons

  1. Performance breakthrough: Reached state-of-the-art results, surpassing RNN-based systems
  2. Parallelization: Processes in parallel, making efficient use of GPUs/TPUs
  3. Long-range dependencies: Solves the problem of distant dependencies in a sequence
  4. Simplicity: A clean design that is easier to understand and implement than complex RNNs
  5. Open source: The community quickly adopted and improved it
  6. Data boom: It arrived just as large datasets became available (Common Crawl, Wikipedia)

Variants and Improvements

BERT (Bidirectional Encoder Representations)

  • Google, 2018
  • Pre-trains deep bidirectional Transformers
  • Successful in question answering and text classification
  • A cornerstone of NLP

GPT Series (Generative Pre-trained Transformer)

  • OpenAI, from GPT-2 to GPT-3 and GPT-4
  • Emphasizes generative ability
  • Pre-trained on huge amounts of text
  • Breakthroughs in text generation, creative writing, and code generation

Transformer-XL and Longformer

  • Address the context length limit of the original Transformer
  • Extend the ability to process long sequences
  • Applied to document-level tasks

Efficient Transformer Variants

  • Reformer, Performer: Reduce the quadratic complexity of self-attention
  • Optimized for very long sequences

Vision Transformers (ViT)

  • Apply the Transformer to computer vision
  • Process an image as a sequence of patches
  • State-of-the-art in image recognition
  • Challenge the dominance of CNNs in vision

Audio and Multimodal Transformers

  • Process audio and speech recognition
  • Multimodal tasks: combining text, image, and audio
  • Versatile as a general-purpose sequence processing architecture

Applications Beyond NLP

  • Computer vision: Image recognition, object detection, image generation
  • Speech recognition: Speech-to-text systems, audio classification, music generation
  • Time series analysis: Forecasting, anomaly detection
  • Drug discovery: Analyzing protein sequences, predicting drug interactions
  • Robotics: Robot control from sequences of sensor data and actions
  • Scientific discovery: Analyzing scientific data, detecting patterns

References

See also: Attention Mechanism · Vision Transformers · RNN And LSTM