nLab singular learning theory

Idea

Singular learning theory applies results from algebraic geometry to statistical learning theory. In the case of learning algorithms, such as deep neural networks, where there are multiple parameter values corresponding to the same statistical distribution, the preimage of the target distribution may take the form of a singular subspace of the parameter space. Techniques from algebraic geometry may then be applied to study learning with such devices.

References

Textbook treatments:

• Sumio Watanabe, Algebraic geometry and statistical learning theory, CRC Press, 2009.

• Sumio Watanabe, Mathematical theory of Bayesian statistics, Cambridge University Press, 2018.

For an informal discussion:

• Jesse Hoogland, Neural networks generalize because of this one weird trick (blog post); Jesse Hoogland, Filip Sondej, Spooky action at a distance in the loss landscape (blog post).