On: Coding Self-Attention, Multi-Head Attention, Cross...

[dlkf]

It’s debatable to what degree ”attention” in LLMs relates to ”attention” in psychology. See Cosma Shalizi’s note on this http://bactra.org/notebooks/nn-attention-and-transformers.ht...