On N-dimensional Rotary Positional Embeddings
One of the simplest ways of encoding relative positional information in attention is to add a scalar to each of the attention logits, with a value somehow depending on the distance between the corresponding query and key (e.g. learned values in T5, fixed values decreasing with distance in ALiBi).
However, this makes it difficult for a query to attend to any specific (key, relative position) pair. In particular, the query must have a component pointing in the direction of the desired key, but this increases the attention scores with all tokens with that key, regardless of their position.
Rotary positional embeddings (RoPE) are an elegant solution to this problem. Essentially, the query and key vectors for every token are rotated by an angle proportional to the token's 1-d coordinate position.
