Given a (oriented) topological manifold $X$, its mapping class group $MCG(X)$ is the group of isotopy classes of (orientation preserving) homeomorphisms $X\to X$.
Often this is considered specifically for $X$ a Riemann surface with punctures in which case a central role is played by Dehn twists.
The mapping class group is of importance in many areas of geometry including study of Teichmüller spaces, of moduli spaces of surfaces, of automorphisms of free groups and in geometric and combinatorial group theory, hyperbolic geometry and so on. Some of the key contributors were Max Dehn, Jakob Nielsen, William Thurston, David Mumford. Recent proof of the related Mumford conjecture has been accomplished by Madsen and Weiss.
For $\mathbf{Aut}(X)$ the automorphism group of the manifold formed in Euclidean topological geometry, hence equipped with its canonical structure of a topological group. Let furthermore $\mathbf{Aut}_0(X) \hookrightarrow \mathbf{Aut}(X)$ be the inclusion of the connected component of the identity.
Then
is the corresponding coset space/quotient group. In other words, the mapping class group is the group of homeomorphism of $X$ onto itself, modulo isotopy.
This is a discrete group. Equivalently it is the group of connected components of $\mathbf{Aut}(X)$. If $X$ is a smooth manifold, then the mapping class group is the group of connected components of the diffeomorphism group
If $X$ is a manifold with boundary $\partial X$, then it is usual to consider automorphisms which restrict to the identity on the boundary.
The mapping class group for 2-dimensional manifolds controls the moduli stack of complex curves.
The classifying spaces of mapping class groups for 2-dimensional manifolds may also be encoded combinatorially in the geometric realization of a category of ribbon graphs. See there for details.
One of the classical results is that the (oriented) mapping class group of the torus $\mathbb{R}^2/\mathbb{Z}^2 \cong (S^1)^2$ is isomorphic to the special linear group $SL_2(\mathbb{Z})$ (more generally, $MCG(\mathbb{R}^n/\mathbb{Z}^n) \cong SL_n(\mathbb{Z})$). Certain generators called Dehn twists may be visualized as cutting a torus along a circle $\{a\} \times S^1$ (or $S^1 \times \{b\}$), thus producing a cylinder, then twisting one of the ends of the cylinder through $2\pi$ and reattaching the two ends.
Another example is a 2-disk with $n$ punctures. The group of diffeomorphisms (fixing the boundary pointwise) modulo isotopy is the braid group $B_n$.
The relation to the homotopy type of the diffeomorphism group is as follows:
For $\Sigma$ a closed orientable surface, then the bare homotopy type of its diffeomorphism group is
if $\Sigma$ is the sphere then
if $\Sigma$ is the torus then
in all other cases all higher homotopy groups vanish:
The first statement is due to (Smale 58), see also at sphere eversion. The second and third are due to (Earle-Eells 67, Gramain 73).
See (Hatcher 12) for review.
The ordinary cohomology with rational coefficients of the delooping of the stable mapping class group of 2-dimensional manifolds (hence essentially the orbifold cohomology of the moduli stack of complex curves) is the content of Mumford's conjecture, proven in (Madsen-Weiss 02).
Surveys include
Massyuyeau, A short introduction to mapping class groups (pdf)
Benson Farb, Dan Margalit, A primer on mapping class groups, draft, web
Nikolai V. Ivanov. Mapping class groups. In Handbook of geometric topology, pages 523–633. North-Holland, Amsterdam, 2002; 1998 draft: ps
Bojko Bakalov, Alexander Kirillov, chapter 5 of Lectures on tensor categories and modular functor (web, pdf)
See also
wikipedia: mapping class group
Ib Madsen, Michael Weiss, The stable moduli space of Riemann surfaces: Mumford's conjecture, Ann. of Math. (2) 165 (2007), no. 3, 843–941, MR2009b:14051, doi, math.AT/0212321
John Harer, The second homology group of the mapping class group of an orientable surface, Invent. Math., 72(2):221–239, 1983; The cohomology of the moduli space of curves in: Theory of moduli (Montecatini Terme, 1985), Lecture Notes in Math. 1337, p. 138–221. Springer, Berlin, 1988.
Robert C. Penner, The decorated Teichmüller space of punctured surfaces, Commun. Math. Phys. 113 (2) (1987) 299–339. MR89h:32044; A construction of pseudo-Anosov homeomorphisms, Trans. Amer. Math. Soc. 310 (1):179–197, 1988.
C.J. Earle, J. Eells, The diffeomorphism group of a compact Riemann surface, Bulletin of the American Mathematical Society 73(4) 557–559, 1967
Allen Hatcher, William Thurston, A presentation for the mapping class group of a closed orientable surface, Topology, 19(3):221–237, 1980.
Allen Hatcher, A 50-Year View of Diffeomorphism Groups, talk at the 50th Cornell Topology Festival in May 2012 (pdf)
Max Dehn, Papers on group theory and topology. Springer-Verlag, New York, 1987. Transl. from German with intro. and appendix by John Stillwell, and appendix by Otto Schreier.
Jakob Nielsen, Untersuchungen zur Topologie der geschlossenen zweiseitigen Flächen. I-III, Acta Math. 50 (1927), no. 1, 189–358, MR1555256, doi; Acta Math. 53 (1929), no. 1, 1–76, MR1555290, doi; Acta Math. 58 (1932), no. 1, 87–167, MR1555345, doi
David Mumford, Abelian quotients of the Teichmüller modular group, J. Analyse Math., 18:227–244, 1967.
David Mumford, Towards an enumerative geometry of the moduli space of curves, Arithmetic and geometry, Vol. II, Birkhäuser Boston, Boston, MA, 1983, pp. 271–328, MR85j:14046
Ulrike Tillmann, On the homotopy of the stable mapping class group, Invent. Math. 130 (1997), no. 2, 257–275, MR99k:57036, doi
S. Morita, Introduction to mapping class groups of surfaces and related groups, in: Handbook of Teichmüller theory (A. Papadopoulos, editor), vol. I, EMS Publishing House, Zürich, 2007, 353–386.
E. Y. Miller, The homology of the mapping class group, J. Differential Geom. 24 (1986), no. 1, 1–14, MR88b:32051, euclid