Mathematics of Attention Mechanisms
July 26, 2025
Yesterday, I left off with the concept behind attention mechanisms. The big picture was that vectors become contextually aware, therefore making the output more meaningful.
Today I learned about about the mathematics behind attention mechanisms, how exactly vectors are made more contextually meaningful through linear algebra. Start out with a sentence, use your tokenizer to represent it with vectors, and map each word to a variable \(x^{k}\)
$$ \text{"Your journey starts with one step"} \;\longrightarrow\; \begin{aligned} \mathbf{x}^{(1)} &= \begin{bmatrix} 0.43 & 0.15 & 0.89 \end{bmatrix} \;\bigl(\text{“Your”}\bigr)\\ \mathbf{x}^{(2)} &= \begin{bmatrix} 0.55 & 0.87 & 0.66 \end{bmatrix} \;\bigl(\text{“journey”}\bigr)\\ \mathbf{x}^{(3)} &= \begin{bmatrix} 0.57 & 0.85 & 0.64 \end{bmatrix} \;\bigl(\text{“starts”}\bigr)\\ \mathbf{x}^{(4)} &= \begin{bmatrix} 0.22 & 0.58 & 0.33 \end{bmatrix} \;\bigl(\text{“with”}\bigr)\\ \mathbf{x}^{(5)} &= \begin{bmatrix} 0.77 & 0.25 & 0.10 \end{bmatrix} \;\bigl(\text{“one”}\bigr)\\ \mathbf{x}^{(6)} &= \begin{bmatrix} 0.05 & 0.80 & 0.55 \end{bmatrix} \;\bigl(\text{“step”}\bigr) \end{aligned} $$
Now that we have our words tokenized, we go onto calculating the new context vector, \(z_2 \). To find it, we need to follow the below mathematical steps, where \((Q, K, V)\) are weighted representation of each vector \(x^k\) (where each weight is found during training). \(\omega_{ij}\rightarrow \alpha_{ij}\) are dot products to normalized dot products of a two words. Dot products are mathematically significant because they represent alignment in space, which allows us to see how close two vectors are. Finally, \(z\) represents the summation of al the words in the sentence. The steps are listed below
$$ Q_i = x_i W^Q, \quad K_j = x_j W^K, \quad V_j = x_j W^V $$ $$ \space $$ $$ \omega_{ij} = Q_i \cdot K_j^T $$ $$ \space $$ $$ \alpha_{ij} = \text{softmax}(\omega_{ij}) $$ $$ \space $$ $$ z_i = \sum_{j=1}^{n} \alpha_{ij} V_j $$
So let’s dive into how we would calculate a context vector for the word ‘journey’, following the above steps. Firstly, we calculate \(\alpha_{2j}\) and apply the softmax, displayed below: $$ \vec{z_2} = \begin{bmatrix} \varnothing \\ \varnothing \\ \varnothing \\ \varnothing \\ \varnothing \\ \varnothing \end{bmatrix} \rightarrow \begin{bmatrix} 0.9544 \\ 1.4950 \\ 1.4754 \\ 0.8434 \\ 0.7070 \\ 1.0865 \end{bmatrix} \rightarrow \text{softmax}(x_i) \rightarrow \begin{bmatrix} 0.1385 \\ 0.2379 \\ 0.2333 \\ 0.1240 \\ 0.1082 \\ 0.1581 \end{bmatrix} $$ These steps are represented by this code:
query = inputs[1]
print(inputs.shape[0])
attention_score_x2 = torch.empty(inputs.shape[0])
for (index, x_i) in enumerate(inputs):
attention_score_x2[index] = torch.dot(x_i, query)
attn_weights_2_solid = torch.softmax(attention_score_x2, dim=0)
Finally, we finish up by calculating that summation and getting our final contextual vector $$ \vec{z_{2}} = \begin{bmatrix} \varnothing \\ \varnothing \\ \varnothing \\ \varnothing \\ \varnothing \\ \varnothing \end{bmatrix} \rightarrow \alpha_{21} V_1 + \alpha_{22} V_2 + \cdots \rightarrow \begin{bmatrix} 0.4371 \\ 0.4371 \\ 0.4371 \\ 0.4371 \\ 0.4371 \\ 0.4371 \end{bmatrix} = z_2 =\sum_j \alpha_{2j}V_j $$
It’s nice to know how these operations work behind the scenes. I find that when I know how something works at this level, it becomes a lot easier to implement it without assistance. In previous attempts to code big complicated projects like this, I’ve always had the problem of getting into a copy-paste loop where I’m just copying from the book’s GitHub. Then, I have no idea how to implement it, how to apply it to my own text, or anything. And this should be a pattern in my career, to deeply understand something.
Now, I’m onto the remaining parts of chapter 3. I’m guessing that this is going to be applied to longer pieces of text, I’m guessing I’m going to be optimizing the strides and whatnot and other things that come with scaling these operations. I’m having a lot of fun with this project, but I need to remember it’s only been 5 days. There are harder concepts to come, there’s going to be a stage of complete tediousness and frustration when it comes to making it a well polished product, and this is all to be expected. I expect to have a hard time with this and I expect that I can get through that phase and produce something impressive
