Apologies for the delay, and thank you for your patience.
> `angles and periodicity become implicit`
Intuitively, RoPE optimizes over trigonometric functions, the derivatives of which are also trigonometric functions, which are (i) not monotonic and (ii) periodic. This *should*, in theory at least, pose optimization issues. In contrast, APE optimizes over (parameterized) orthogonal matrices, the derivatives of which should be better behaved.
> `insights`
We have conducted the experiment you requested (*i.e.*, turning RoPE trainable naively and, *vice versa*, freezing APE). To our dismay, we found that:
* RoPE diverged only twice across experiments.
* In most cases, a frozen APE outperformed a trained APE.
* In most cases, a trained RoPE outperformed a frozen APE.
To provide an empirical correspondence between RoPE and APE, as requested, we then devised a back-and-forth algorithm between the two in the sequential case. We detail the process below.
### **Parameterizing APE**
First, our parameterization of orthogonal primitives follows the standard matrix exponentiation trick on skew symmetric matrices:
1. Start from an upper triangular square matrix $A$. (your trainable parameter)
2. Obtain a skew symmetric $B := A - A^\top$.
3. Take the matrix exponent $C := e^{B}$. $C$ is orthogonal.
### **RoPE to APE**
To go from RoPE to APE:
1. Start from a sequence of RoPE angles $\Theta := [\theta_1, \theta_2, \theta_3, \dots, \theta_n]$.
2. Block-diagonalize $\Theta$ into
$$
C := \begin{bmatrix}
cos\theta_1 & -sin\theta_1 & 0 & 0 & \dots \\\\
sin\theta_1 & cos\theta_1 & 0 & 0 & \dots \\\\
0 & 0 & cos\theta_2 & -sin\theta_2 & \dots \\\\
0 & 0 & sin\theta_2 & cos\theta_2 & \dots \\\\
\vdots & \vdots & \vdots & \vdots & \ddots
\end{bmatrix}
$$
3. Stop here if not interested in parameterization. Otherwise, obtain the matrix logarithm of $C$, $ B:= \mathrm{log}(C)$ using a numerical solver.
4. Find $A$ such that $\mathrm{MSE}(B, A - A^\top)$ is minimized. $B$ is not positive-definite so it does not admit a Cholesky decomposition -- this means you have to use an iterative approximation. We find it's easy to obtain one with minimal error within seconds using gradient-based optimization.
### **APE to RoPE**
To go from APE to RoPE (this is a practical variant of what's described in Section 3.2)
1. Start from orthogonal matrix $C$.
2. Perform the polar decomposition $C = UP$. Since $C$ is invertible, $U$ is unique, and denotes a rotation/reflection along the axes specified by $P$. For $P \equiv 1$, $U$ is block-diagonal and corresponds to $C$ above (*i.e.* the absolute angles of its eigenvalues correspond to $\Theta = [\theta_1, \theta_2, \dots]$).
### **Results**
Besides explicating the relation between APE and RoPE, this back-and-forth allows us to do something more: we can now *initialize* APE with a parameterization that produces the angles used by RoPE! Doing so allows us to examine the effects of trainability *in juxtaposition with initialization*. We find that in almost every setup a trainable APE initialized as RoPE **outperforms all other models**. Together, these results empirically confirm both **your claim** that RoPE can do better if trained, and **our claims** that (i) gains are also due to initialization, and (ii) RoPE has been "tuned" offline.
For the sake of comparison, we summarize all results in the table below. `APE`: Algebraic (seq), `init`: RoPE-like initialization, `train`: trainable. `[b]`: breadth-first, `[d]`: depth-first. We do not include the 2 diverging RoPE runs so as not to obfuscate the results.
| **Task** | APE[init,train] | RoPE[train] | APE | APE[train] | RoPE |
| ----------- | ------------------ | --------------- | ------------- | --------| -----------------|
| Copy | **1** | **1** | **1** | **1** | **1** |
| Reverse | **1** | **1** | **1** | **1** | **1** |
| Repeat | **1** | **1** | **1** | **1** | **1** |
| Tree-Copy [b] | **1** | **1** | 2.2 | **1** | 1.9 |
| Tree-Copy[d] | **1.3** | 1.8 | 2.2 | 2.4 | 3.2 |
| Rotate[b] | **2.9** | 4.2 | 3.4 |5.2 | 4.9 |
| Rotate[d] | **2.3** | 2.5 | 3.4 |5.7 | 6.6 |
| C3[b] | **1.1** | 1.2 | 1.2 |1.5 | 2.0 |
| C3[d] | **1.9** | 1.9 | 2.1 |2.3 | 2.4 |
| OP[b] | 2.3 | 2.5 | 2.6 | **1.8** | 2.6 |
| OP[d] | 20.0 | 21.3 | 19.9 | 29.3 | 41.2 |
So to summarize: APE[init, train] > RoPE[train] > APE > APE[train] > RoPE
---
To us this seems like a very interesting find. Does this help answer your questions?
In any case, many thanks for encouraging us to do these additional experiments. These empirical findings have helped us consolidate the effect of training and initialization for both RoPE and APE, as well as further reinforce the empirical strengths of Algebraic encodings.