This entry (apart from a few remarks) will be mainly about general statistics and mathematical statistics. For more about physical applications in statistical mechanics and probabilistic interpretation of quantum mechanics, see there. There is also a technical notion of statistic (singular).

Idea

Statistics studies the analysis of collections of random or sample data, and the probabilistic likelihood of various inferences on the basis of these data, as well as the mathematical regularities in large ensembles of occurrences of such data.

In physics, statistics also pertains to the behaviour of large ensembles of particles. For identical particles, this is the subject of particle statistics and for general systems the subject of statistical mechanics.

Mathematical statistics is based on probability theory. Most of the standard formalism uses measure theory as used in probability. Statistical mechanics in addition heavily uses ergodic theory.

Michael Schmitt, Statistics for theorists, quick 3-lecture intro for theoretical physicists at TASI 2020 (each around 1 and half hours) lec1 mp4 slides pdf, lec2 mp4pdf, lec3 mp4pdf

Peter McCullagh, What is a statistical model?, Ann. Statist. 30:5 (2002), 1225-1310 euclidMR1936320doi – on applying category theory to describe statistical models.

Lior Pachter, Bernd Sturmfels, Tropical geometry of statistical models, PNAS 101 no. 46, 16132–16137, doi