Within the nLab, “nice category of spaces” is a general but inexact term referring to nice or convenient properties one would like a category of spaces to have for some purpose (“space” here connoting something along topological lines), but typically not satisfied by the category of topological spaces and continuous maps itself, thus necessitating a move to some nicer category to suit one’s purposes.
The category Top of topological spaces lacks many good categorical properties. It is complete and cocomplete, and is extensive, but:
The lack of cartesian closure and, to a lesser extent, local presentability, is especially problematic for homotopy theory. Many different solutions for repairing lack of cartesian closure have been proposed, generally involving either restricting to a subcategory of Top (usually reflective or coreflective, so that it inherits completeness and cocompleteness), enlarging it to a supercategory, or some combination thereof. Most involve restricting the topologies to those that can be specified on “small” (and in particular, compact) subsets.
In particular, a convenient category of topological spaces is, in the technical sense of the nLab, a cartesian-closed category of spaces together with some other useful properties (q.v.).
The frequent choice among algebraic topologists today is to use the subcategory of compactly generated spaces, which is cartesian closed, but not locally cartesian closed. It is a coreflective subcategory of a reflective subcategory of $Top$.
Homotopy theorists often find the category of simplicial sets to be an especially nice environment. It is for example a Grothendieck topos, thus a locally cartesian closed category and satisfying all exactness properties one expects of toposes, and is a locally presentable category. The fact that every topological space has a simplicial set as its singularization? then becomes an application of the homotopy theory of simplicial sets to the study of topological spaces, rather than a way to use simplicial sets to study the homotopy theory of topological spaces.
For more on this see Top, homotopy theory and infinity-groupoid.
The subcategory of Delta-generated spaces, recently proposed by J. H. Smith, is also both cartesian closed and locally presentable.
An approach of mainly historical interest is to use quasitopological spaces, an enlargement of $Top$ which is cartesian closed.
The category $PsTop$ of pseudotopological spaces (also called Choquet spaces) is a quasitopos containing $Top$ as a full reflective subcategory. In particular, $PsTop$ is locally cartesian closed (but not locally presentable).
In his paper On a topological topos, Peter Johnstone described a Grothendieck topos $E$ which contains the category of sequential topological spaces as a full reflective subcategory which is closed under many colimits (including all those used to define CW complexes). Again, since $E$ is a Grothendieck topos, it is locally presentable and locally cartesian closed. Moreover, the geometric realization and singular complex functors form a geometric morphism between $E$ and the category of simplicial sets. The “underlying set” functor $E\to Set$ is not faithful, but it is faithful on the full subcategory of subsequential spaces, which contain the sequential spaces and form a quasitopos. See topological topos.
The category of compact Hausdorff spaces is perfectly nice for some purposes. While neither cartesian closed nor locally presentable, it is however a complete and cocomplete pretopos. In this way it is both a category of nice topological spaces and a nice category of topological spaces, thus an exception proving the “rule” described at dichotomy between nice objects and nice categories.
The category of locales and the full subcategory of sober spaces can be considered nice for certain purposes. The category of locales is extensive and is opposite to the category of frames (which is monadic over $Set$ and thus exact). The category of locales is however neither cartesian closed nor locally presentable, although there is a nice description of exponentiable locales. Johnstone’s Stone Spaces gives an account of topology via locale theory.
John Milnor proposed the category of spaces having the homotopy type of a CW complex as a nice category of spaces. If $X$ and $Y$ are objects and $X$ is compact, then there is an exponential object $Y^X$ in this category. This by the way is also a category of nice topological spaces.
The category of algebraic lattices, considered as a full subcategory of $T_0$-spaces, is a nice cartesian closed category of spaces in which to do domain theory. Related to this is the category of equilogical spaces, which is locally cartesian closed (and thus also regular) and arises as the reg/ex completion of the category of $T_0$ spaces.
Peter May, A Concise Course in Algebraic Topology (Chapter 5, for compactly generated spaces)
O. Wyler, Convenient categories for topology
L. Fajstrup and J. Rosicky, A convenient category for directed homotopy (for Delta-generated spaces)
E. Spanier, Quasi-topologies (for quasi-topological spaces)
O. Wyler, Lecture notes on topoi and quasitopoi (for pseudotopological spaces)
J. Milnor, On spaces having the homotopy type of a CW-complex, Trans. Amer. Math. Soc. 90, no. 2 (1959), 272-280.