ZC-Swish: Stabilizing Deep BN-Free Networks for Edge and Micro-Batch Applications

The core insight is not just that ZC-Swish stabilizes training, but that it does so by enforcing zero-centered outputs, which preserves gradient flow through deep networks in the same way batch normalization does statistically, without requiring large batch sizes to compute reliable statistics. That distinction matters because the problem is not activation choice alone but the statistical coupling BN creates between samples in a batch.

The stability theme here connects directly to two recent papers in the archive. The nonlinear separation principle paper from April 16 ('A Nonlinear Separation Principle') also tackled training stability through structural constraints, specifically deriving conditions under which monotone activations keep recurrent networks within a stable weight space. ZC-Swish approaches the same class of problem from the opposite direction: instead of constraining weights, it constrains the activation output distribution. The medical imaging angle also echoes 'SegWithU' from April 16, which addressed single-forward-pass inference under resource constraints, a regime where micro-batch training is a direct upstream dependency.

The real test is whether ZC-Swish holds up in federated learning benchmarks on heterogeneous data splits, where batch statistics are not just small but systematically biased across clients. If published follow-up results show stable convergence under non-IID partitioning, the claim generalizes; if they only replicate the homogeneous micro-batch case, the contribution is narrower than framed.