Interactive lab · 2-D minibatch SGD

Where do the iterates go?

Draw a Gaussian dataset once. Each step picks a minibatch and multiplies by $I-\eta H_t$. Move $\eta$ and $B$, then press Run. The cloud is the iterate law; the left table says whether the linear map shrinks or grows.

$$x_{t+1}=(I-\eta H_t)\,x_t-\eta\Big(\tfrac{c_{4,t}}{6}\|x_t\|^2+\tfrac{c_{6,t}}{120}\|x_t\|^4\Big)x_t+\sigma\,\xi_t, \qquad H_t=\frac1B\sum_{i\in\mathcal B_t}z_i z_i^\top, \quad z_i\sim\mathcal N(0,\Sigma),\quad \Sigma=\mathrm{diag}(\mu_1,\mu_2)$$

$B=N$: full batch, $H_t$ is fixed. $B<N$: $H_t$ changes every step (matrices usually do not commute). $c_4,c_6$ can keep the cloud bounded even when the linear map expands. Color bar on $\eta$: teal = mean-square stable, then Lyapunov edge, then past it.

Presets

Same data, different $B$. Click one, then watch the cloud.

Step size

η 2.500

mean-square stable to Lyapunov edge to $K_{-1/2}=1$ past the edge 2/λ_max

Data & batch

dataset N

batch knob B398

exact B

Σ eig. μ₁

Σ eig. μ₂

One fixed dataset $z_i\sim\mathcal N(0,\mathrm{diag}(\mu_1,\mu_2))$. $B=N$ is full batch.

Stochastic self-stabilization

mean c̄₄

std. s₄

mean c̄₆

std. s₆

noise σ

drift clip

Nonlinear pull that can bound the cloud. $\sigma$ is extra noise.

Simulation

particles

steps T

burn-in

seed

Linear map

2/λ_max
Lyapunov χ
mean-square ρ
K_−1/2
edges η_MS, η_L, η_−1/2
mean batch eigs
commutator
mean ‖M₄‖
P(‖M₄‖ > 1)
B / N
Hessians

—

Harmonic stability

$H(h)=\mathbb{E}\big[|1-h\,\lambda_{1,B}|^{-1}\big]$ along the top batch eigenvalue. $H>1$ still returns mass to $0$. Harmonically unstable when $H<1$.

H(h)
K_−1/2
h_H (crossing)

—

Iterate cloud p(x₁, x₂)

marginal p(x₁)

fixed axes
not rotating with H_t

x₁ → ↑ x₂

p(x₂)