Fourier Transform Networks: Boundary-Aware Spectral Sequence Modeling Without Attention

Model

Overall architecture

FTN is a causal language model with standard token embeddings, positional embeddings, a stack of FTN blocks, a final normalization layer, and a tied output projection. The core novelty is the FTN block itself.

Each FTN block processes a hidden sequence through two parallel branches:

• a local branch, designed to capture short-range dependencies through causal convolution with a limited receptive field;
• a global branch, designed to capture sequence-wide structure through causal FFT convolution.

Both branches are fused, added back through a residual connection, and followed by an MLP sublayer.

Code-level configuration

The implementation used in this paper is defined directly in modeling_ftn.py. The FTN configuration object contains

along with the boundary-specific parameters boundary_edge_width, boundary_edge_mix, and boundary_center_mix. In the code, $d$ corresponds to d_model, $N$ to num_layers, $d_{\mathrm{ff}}$ to ffn_dim, $K_{\ell}$ to the local window size, and $K_g$ to the global kernel size.

At the top level, the model computes

then applies $N$ FTN blocks, followed by

with tied token embedding weights when tie_word_embeddings=True. Training uses standard next-token cross-entropy on shifted logits.

Causal FFT convolution

The FTN global branch replaces attention with a causal convolution implemented in the frequency domain. In code, the tensor is transposed from shape $[B,L,d]$ to $[B,d,L]$, padded implicitly by choosing

and transformed channel-wise with a real FFT. Given an input sequence $x \in \mathbb{R}^{B \times L \times d}$ and a learned depthwise kernel $w \in \mathbb{R}^{d \times K}$, FTN computes

and then

with an optional channel-wise bias added afterward. The truncation to the first $L$ positions is exactly the code path that enforces causal linear convolution rather than circular wraparound. The implementation also includes a direct-reference convolution path used for numerical preflight checks:

with grouped convolution over channels. This gives global receptive fields with $O(L \log L)$ spectral mixing instead of $O(L^2)$ attention.

When use_frequency_gain=True, the code multiplies by a learned positive gain tensor

so that

This is the exact mechanism used by the v2_boundary global branch to reshape the spectral profile beyond the kernel alone.

From v1_add to v2_boundary

The early FTN variant, v1_add, used the same broad dual-branch idea but proved weak in both loss and qualitative generation. Diagnostics suggested that the global branch was often oversmoothed, with upper layers concentrating almost all energy in low-frequency bands.

The v2_boundary variant introduces three key changes:

1. Branch-specific input and output projections. Both local and global branches receive learned pointwise projections before and after convolution, allowing each branch to shape its own representational subspace.
2. Learned global frequency gains. The global FFT branch applies a learned positive gain over frequency bins, allowing the model to upweight or downweight different spectral regions rather than inheriting a fixed spectral bias from the kernel alone.
3. Boundary-aware residual gating. The global branch mixes its FFT output with a learned residual projection using a position-dependent gate table. Edge positions are initialized to trust the residual path more than the spectral path, while central positions are initialized to trust the spectral path more strongly.

The actual FTN block implemented in code is close to the following equations. Let $h$ denote the layer-normalized input:

The local branch is

The global branch is

For v2_boundary, the code then slices a learned table of position logits,

applies a sigmoid gate,

and mixes FFT and residual paths elementwise:

The normalized global branch is

The boundary gate is initialized asymmetrically by position. If $w_e$ is the edge width, then positions in the first and last $w_e$ tokens are initialized toward boundary_edge_mix, while the center positions are initialized toward boundary_center_mix. In the implementation used here, those defaults are 0.2 at the edges and 0.8 in the middle. This was specifically introduced to reduce brittle edge behavior and avoid treating causal sequence modeling as if it were naturally periodic.

Fusion

We evaluated three fusion schemes for combining the local and global branches:

• additive fusion,
• concatenation followed by projection,
• learned gated fusion.

Unexpectedly, additive fusion proved most effective in our setting. This result suggests that once the branches themselves are well-formed, more complex routing may not be the main bottleneck. We return to this observation in Section discussion.

More precisely, the fusion module in code accepts one of three modes:

In all cases, the result is passed through an output projection. The block then applies residual addition and a feed-forward network:

Implementation-faithful block summary

To make the connection to the actual model code explicit, one FTN block in modeling_ftn.py is structurally:

hidden = input_norm(x)
local_hidden = local_in_proj(hidden)
local_branch = local_norm(local_out_proj(local_conv(local_hidden)))
global_hidden = global_in_proj(hidden)
global_fft_out = global_out_proj(global_conv(global_hidden))
gate = sigmoid(boundary_gate_table[:seq_len])
global_branch = global_norm(gate * global_fft_out
                            + (1 - gate) * global_residual_proj(hidden))
fused = fusion(local_branch, global_branch)
x = x + residual_dropout(fused)
x = x + mlp(mlp_norm(x))

This is not merely an implementation detail; it defines the actual object of study in this paper. In particular, the learned frequency gains, boundary gate table, and residual-mixing path are all present in the deployed code and in the published checkpoints.