Weight Implmentation, Self Attention
July 27, 2025
In our previous walkthrough of creating attention mechanisms, we left out computations of \(W^{Q}, W^{T}, W^{V}\). This would be the final step in making a basic SelfAttention mechanism.
As with all weights encountered so far in LLM, we initialize them randomly. W_query = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False) is a replicable example of how we would initialize \(W^{Q}\). Now, using this new notation which I learned today, we can use torch’s @ notation in order to compute mass matrix multiplication in order to find the \((Q,K,V)\). The following x_j @ W_query, x_j @ W_key, … computes those variables and stores them in a tensor corresponding to each word in the sentence: $$Q_i = x_i W^Q, \quad K_j = x_j W^K, \quad V_j = x_j W^V$$
Last time took a while to get through all the steps with LaTex and whatnot, so I’m going to be brief. We basically include weights in all computations leading up to the context vectors. After that, we created a class to do those computations
class SelfAttention_v1(nn.Module):
def __init__(self, d_in, d_out):
super().__init__()
self.W_query = nn.Parameter(torch.rand(d_in, d_out))
self.W_key = nn.Parameter(torch.rand(d_in, d_out))
self.W_value = nn.Parameter(torch.rand(d_in, d_out))
def forward(self, x): #x is an entire input tensor
keys = x @ self.W_key
queries = x @ self.W_query
values = x @ self.W_value #access the first by values[0], same for above
attn_scores = queries @ keys.T
attn_weights = torch.softmax(
attn_scores / keys.shape[-1]**0.5, dim=-1
)
context_vec = attn_weights @ values
return context_vec
After that, I learned another more robust implementation of the attention mechanisms using torch’s linear functions. Simple fix, not really worth putting down.
The last thing I learned today was called ‘casual attention’. If you consider what we’ve been doing, we’ve been taking an entire sentence as input and creating contextual vectors based on that. However, LLM’s produce words one at a time — they don’t have access to the future of the sentence as they are writing it, just like a LLM couldn’t predict BOO! They would expect that. So we introduce the concept of casual attention, where we mask the future context vectors so that the current word can only predict based on words behind it. I was just suspicious, saying that “well, wouldn’t the LLM need to have access to the future of the sentence in order to mask them to begin with?” Yeah I thought that too but only for about 2 seconds, because I realized that this is probably done during the training process to ensure that, in the real situation where it will definitelyy not have access to the future results, it car reproduce it. Just like doing a practice test without looking at the answers. I’ll paste a ‘masked’ array of normalized attention scores to visualize. $$ \begin{bmatrix} 1.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\ 0.4902 & 0.5098 & 0.0000 & 0.0000 & 0.0000 & 0.0000 \\ 0.3256 & 0.3379 & 0.3365 & 0.0000 & 0.0000 & 0.0000 \\ 0.2429 & 0.2497 & 0.2492 & 0.2582 & 0.0000 & 0.0000 \\ 0.2008 & 0.1982 & 0.1978 & 0.2075 & 0.1957 & 0.0000 \\ 0.1582 & 0.1664 & 0.1659 & 0.1727 & 0.1597 & 0.1771 \end{bmatrix} $$
I’m not that satisfied with today. I felt really good in the morning, but after the first session, I kinda ran out of energy and couldn’t work anymore. I wonder what it is. Is it just a discipline problem, or is actually my energy? I think I’m going to try something out tomorrow. We can always improve. I’ll try better desk hygiene, as in not fidgeting as much, not getting up and getting water, staying focused while I’m there. Paying the fee. Besides it’s not supposed to be easy all the time, yeah. So tomorrow I’ll do it all over again. I’ll plan right after this.
On the bright side, I finally started reaching out to some important people and took notes on important opportunities. I should make this a habit, even if it’s like a couple emails a day. I could get some great opportunities. On the dark side, I’m having a little trouble with these tree problems, but that’s expected. Eventually it will be understood it’s just a matter of time.
Tomorrow, I stay home. I like my desk.
