No posiitonal embeddings

[takeraparterer]

In https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb there are no positional embedding whatsoever, which are needed for causal attention like this to function at all

[takeraparterer]

Update:

q, k, v = [self.ln1(x) for _ in range(3)]
attn_out, _ = self.attn(q, k, v)

not how you do it!