Recurrent Looped Transformer: Decoding with Growing Depth per Token

Looped Transformer is currently one of the most discussed architectural ideas.

The new technical report proposes to stretch the cycle already between tokens.

Recurrent Looped Transformer makes the decoder recurrent for each token, including both the prompt and the response.

The causal encoder builds a shared KV memory. For each new token the decoder combines:

> representation of this token from the encoder
> its own final hidden state from the previous token
> cache of recent activations with a sliding window

In a 48‑layer decoder, the computation path after t tokens already passes through 48 t decoder blocks, although each individual token still executes a fixed number of blocks.

Thus, the computation depth grows with the sequence length, while the cost of processing a single token remains constant.

The same state transition is used for pretraining, SFT, generation, and replay in RL. At the boundary between the prompt and the response, the state is not reset.

During RL replay, states are rebuilt from the current model weights instead of reusing stale states from previous runs.

So far this is just an architectural proposal. The author explicitly states that the gains in reasoning, hardware acceleration, and RL scaling are goals that have not yet been measured.

Article: yifanzhang-pro/recurrent-looped-tranformergithub.com/yifanzhang-pro/recurrent-looped-tranformer