11/01/2026

Stability of Dynamical Systems, Residual Streams, and DeepSeek's mHC

Several years ago I was a math professor. My research was stability of dynamical systems: when does a system stay bounded — and when does it blow up, oscillate, or collapse to zero?

LLMs have the same failure modes.
As you stack layers, signals and gradients can vanish (no learning) or explode (training gets destroyed).

Diagram of Hyper-Connections: multiple parallel residual streams mixed by a learned matrix at each layer

Diagram of a residual connection: identity path plus learned residual correction

That’s why residual connections were such a breakthrough:

x ↦ x + F(x)

Instead of forcing every block to learn the full transformation, you keep an identity path and only learn the residual “correction”. The identity route is what keeps information and gradients alive across depth.

Transformers kept this idea: a residual stream that survives hundreds of layers.

Hyper-Connections (HC) ask: what if that residual stream is wider?
Instead of 1 stream, keep n parallel streams and learn how to mix them each layer with a matrix H (so streams can exchange information).

The catch is stability.
Across depth you don’t apply one mixer — you apply a product of mixers:

(... H^(l+2) H^(l+1) H^l) x

Products of unconstrained matrices are where stability goes to die. DeepSeek measured this directly: HC could amplify signals by ~3000× and trigger loss/gradient spikes.

Their fix (mHC) is beautifully “systems”:
constrain the residual mixing matrix so it can redistribute signal, but can’t create or destroy it.

They make H doubly stochastic:

all entries ≥ 0
every row sums to 1
every column sums to 1

Two key consequences:

Non-expansive mixing: ||H||₂ ≤ 1, so the residual path can’t act as an amplifier
Compositional stability: the product of doubly stochastic matrices is doubly stochastic, so the guarantee survives arbitrary depth

In practice: parameterize an unconstrained matrix, make it positive (via exp), then run Sinkhorn–Knopp (iterative row/column normalization) to project onto the Birkhoff polytope (convex hull of permutation matrices).

The payoff shows up at scale (27B): mHC trains stably where HC spikes, and it improves reasoning benchmarks.

And the part GPU/systems folks will appreciate: HC is memory-hungry (more reads/writes).
DeepSeek makes it viable with kernel fusion, selective recomputation, and pipeline/stream overlap — bringing overhead down to ~6.7% at n=4.

This doesn’t feel like an isolated trick. Constraining key weights to “good” mathematical sets (Stiefel, Birkhoff, …) is turning into a theme: constraints as enablers for scale, not just “regularization”.

Paper (mHC): https://arxiv.org/abs/2512.24880
Hyper-Connections (Zhu et al., 2024): https://arxiv.org/abs/2409.19606
Related: https://thinkingmachines.ai/blog/modular-manifolds/