>[!quote] In a Nutshell >Transformer models, introduced in Vaswani et al.'s 2017 paper "Attention Is All You Need," revolutionized NLP and other #domain/ml areas by enabling parallelized processing and capturing long-range dependencies effectively. --- #### General Architecture - **[[Tokenization|Tokenizers]]**, which convert text into tokens. - **Embedding** layer, which converts tokens (see [[Word Embedding]]) and positions of the tokens (see [[Positional Encoding]]) into vector representations. - **Transformer** layers, which carry out repeated transformations on the vector representations, extracting more and more linguistic information. These consist of alternating attention and feedforward layers. There are two major types of transformer layers: **encoder** layers and **decoder** layers, with further variants. - **Un-embedding** layer, which converts the final vector representations back to a [[Probability Distribution]] over the tokens. ![[Pasted image 20250330180455.png|center|400]] | Component | Dimensions | Example Count in GPT3 | | --------------- | :--------: | :-------------------: | | Input Embedding | | $617.558.016$ | | Query | | $14.495.514.624$ | | Key | | $14.495.514.624$ | | Value | | $14.495.514.624$ | | Output | | $14.495.514.624$ | | Up Projection | | | | Down Projection | | | | Unembedding | | $617.558.016$ | >[!brainwaves] Why was this so revolutionary ? >Transformer replace sequential recurrence of [[Recurrent Neural Network Layers|RNNs]] with **parallelizable** attention, enabling efficient training on long sequences (more efficient and avoids vanishing [[Derivative, Gradient, Jacobian and Hessian|gradient]]). > >Additionally, each token can dynamically attend to relevant context at multiple abstraction levels (via multi-head attention). --- #### Components Encoder and decoder differ in their input, the first takes the prompt while the decoder uses previously generated tokens (or targets during training). Additionally, the decoder uses the output of the encoder as context and has to apply masking. ##### Encoder Blocks >[!info] Input Embeddings and Positional Encoding >The initial input is a [[Set|set]] of $n$ tokens $\mathbf{X} = [x_1, x_2, ..., x_n]$, each of which is converted to a $d_{\text{model}}$-dimensional [[Vector Space|vector]] embedding$\mathbf{E} = \text{Embedding}(\mathbf{X}) \in \mathbb{R}^{n \times d_{\text{model}}},$see **[[Word Embedding]]** and **[[Positional Encoding]]** for more information. The input to the model is then$\mathbf{Z} = \mathbf{E} + \text{PE} \in \mathbb{R}^{n \times d_{\text{model}}}.$ >[!info] Multi-Headed Self-Attention >For each input $\mathbf{Z}$, we compute the attention score $\text{head}_i =\text{Attention}(\mathbf{Z}\mathbf{W}_i^Q, \mathbf{Z}\mathbf{W}_i^K, \mathbf{Z}\mathbf{W}_i^V)= \text{softmax}\left(\frac{\mathbf{Q}_i\mathbf{K}_i^\top}{\sqrt{d_k}}\right)\mathbf{V}_i \in \mathbb{R}^{n \times d_v}$for each head $i \in \{1,...,h\}$ in parallel. For detailed information, see **[[Attention Mechanisms]]**. The outputs are then concatenated and projected back into a common [[Latent Variable Models|latent]] [[Space|space]]$\text{MultiHead}(\mathbf{Z}) = \text{Concat}(\text{head}_1, ..., \text{head}_h)\mathbf{W}^O \in \mathbb{R}^{n \times d_{\text{model}}}, \quad \mathbf{W}^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}.$ >[!info] Feed-Forward Layers >Each output vector of the multi-headed attention layer undergoes a standard feed-forward transformation$\text{FFN}(\mathbf{x}) = \max(0, \mathbf{x}\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2,$with $\mathbf{W}_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$, $\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$. >[!info] Residual Connections and Layer Norms >Each layer, consisting of one attention process followed by a couple feed-forward layers has a residual connection and layer normalization$\mathbf{Z}_{\text{out}} = \text{LayerNorm}(\mathbf{Z} + \text{SubLayer}(\mathbf{Z})),$where$\text{LayerNorm}(\mathbf{x}) = \gamma \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta,$with [[Expectations|mean]] $\mu$, [[Covariance and Variance|variance]] $\sigma$ and tunable parameters $\gamma, \beta$. ##### Decoder Blocks >[!info] Input Embeddings and Positional Encoding >The initial input is a [[Set|set]] of $m$ previously generated or target tokens $\mathbf{\tilde{X}} = [\tilde{x}_1, \tilde{x}_2, ..., \tilde{x}_m]$, each of which is converted to a $d_{\text{model}}$-dimensional [[Vector Space|vector]] embedding$\mathbf{E} = \text{Embedding}(\mathbf{\tilde{X}}) \in \mathbb{R}^{m \times d_{\text{model}}},$see **[[Word Embedding]]** and **[[Positional Encoding]]** for more information. The input to the model is then$\mathbf{Y} = \mathbf{\tilde{E}} + \tilde{\text{PE}} \in \mathbb{R}^{m \times d_{\text{model}}}.$ #trash --- Finalize the following - Multi-Headed Self-Attention is the same >[!info] Multi-Headed Cross-Attention >For each input $\mathbf{Y}$, we compute the attention score $\text{head}_i =\text{Attention}(\mathbf{Y}\mathbf{W}_i^Q, \textcolor{red}{\mathbf{Z}}\mathbf{W}_i^K, \textcolor{red}{\mathbf{Z}}\mathbf{W}_i^V)= \text{softmax}\left(\frac{\mathbf{Q}_i\mathbf{K}_i^\top}{\sqrt{d_k}}\right)\mathbf{V}_i \in \mathbb{R}^{n \times d_v}$for each head $i \in \{1,...,h\}$ in parallel. For detailed information, see **[[Attention Mechanisms]]**. The outputs are then concatenated and projected back into a common [[Latent Variable Models|latent]] [[Space|space]]$\text{MultiHead}(\mathbf{Z}) = \text{Concat}(\text{head}_1, ..., \text{head}_h)\mathbf{W}^O \in \mathbb{R}^{n \times d_{\text{model}}}, \quad \mathbf{W}^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}.$ - Multi-Headed cross-attention - Use Keys and values from encoder, queries from decoder - Use Masking during training - Linear, Resicual and Norm are the same ToDo --- Finalize above ##### Output >[!info] Output Layer >The decoder output has to be projected back into the vocabulary space of dimension $V$ via$\mathbf{P} = \text{softmax}(\mathbf{Z}_{\text{decoder}} \mathbf{W}_{\text{out}}),$where $\mathbf{W}_{\text{out}} \in \mathbb{R}^{d_{\text{model}} \times V}$. The next [[Tokenization|token]] can then be chosen via [[Maximum Likelihood Estimator|maximum likelihood]], beam search or sampling. --- #### Training and Inference During training, the encoder processes the full input once and the decoder uses [[Training Sequence Models|teacher forcing]]—receiving the actual target sequence (shifted right) to generate its outputs. During inference, the encoder remains the same, but the decoder generates tokens one by one, feeding its own previous predictions back as input for the next step. Objective To train the above model, we can compare the predicted probabilities $\mathbf{P}$ with true target tokens $\mathbf{Y}$ via the [[Cross Entropy|cross-entropy]]$\mathcal{L} = -\sum_{i=1}^n \sum_{k=1}^V \mathbf{Y}_{i,k} \log \mathbf{P}_{i,k}.$