GPT
GPT
Generative Pre-trained Transformer (GPT) is a decoder-only Transformer model. It is pre-trained on text corpora to predict the next token given preceding context. It uses causal attention masks to ensure tokens only attend to previous positions during training, enabling rapid auto-regressive generation.
Complexity Profile
| Case | Complexity |
|---|---|
| Best Case | O(N^2 * D) |
| Average Case | O(N^2 * D) |
| Worst Case | O(N^2 * D) |
| Space Complexity | O(N^2) |
Code Implementation
# Conceptual block of Causal/Masked Self-Attention in GPT
# The causal mask is a lower-triangular matrix of ones
def causal_attention_mask(seq_len):
# Returns a mask matrix where upper-triangular index weights are -inf
mask = torch.triu(torch.full((seq_len, seq_len), float('-inf')), diagonal=1)
return maskReal-World Applications
- Generative chat bots and code completion services (Copilot).
- Synthetic dataset generation workflows.
- Text summarization, translation, and general instruction following.
Attention
Scaled Dot-Product Attention calculates relevance scores between sequence elements. It takes Query (Q), Key (K), and Value (V) matrices, computes dot products of Q and K, scales them by the square root of query dimension, applies Softmax to yield attention weights, and multiplies the weights with V.
Transformers
Transformers are sequence models introduced in 'Attention Is All You Need'. They discard recurrence and convolutions entirely, relying on Multi-Head Self-Attention layers and Position-wise Feed-Forward Networks. They process sequences in parallel, enabling rapid training on massive web datasets.