Let $(R, \lt)$ be a dense? linear order without endpoints?. An order-minimal or o-minimal structure on $R$ is a structure $\mathcal{S}$ on $R$ such that
The relation $\lt$ belongs to $\mathcal{S}_2$;
The elements of $\mathcal{S}_1$ are precisely finite unions of points and intervals in $R$.
Here an interval can mean a set of the form $I_{a, b} = \{x \in R: a \lt x \lt b\}$, or $I_{\downarrow a} = \{x \in R: x \lt a\}$, or $I_{\uparrow a} = \{x \in R: a \lt x\}$.
A structure on a set $R$ can be thought of as the collection $\mathcal{S} = \bigcup_n \mathcal{S}_n$ of sets that are definable with respect to a one-sorted first-order language $L$ with a given interpretation in $R$. Thus $\mathcal{S}_n$ is the collection of subsets of $R^n$ which are defined by $n$-ary predicates in $L$. The definition of o-minimal structure supposes that $L$ contains a relation symbol $\lt$, and that $\lt$ is interpreted in $R$ as a dense linear order without endpoints.
The o-minimality condition places a sharp restriction on which subsets of $R$ can be defined in the language. Essentially, it means that the only definable subsets of $R$ are those which are definable in terms of constants and the predicates $\lt$ and $=$.
The archetypal example of an o-minimal structure is that of semi-algebraic sets defined over $\mathbb{R}$ (which form a structure due to the Tarski-Seidenberg theorem).
Quite remarkably, quite a lot can be said about the structure of definable sets in an o-minimal structure over $\mathbb{R}$, and this is a very active area of model theory. The notion of o-minimal structure has been proposed as a reasonable candidate for an axiomatic approach to Grothendieck’s hoped-for “tame topology” ([topologie modérée]).
A theory is o-minimal if every model $M$ of $T$ is an o-minimal structure.
Lou van den Dries, Tame topology and O-minimal structures, London Math. Soc. Lecture Notes Series 248, Cambridge U. Press 1998.
Alexandre Grothendieck, Esquisse d’un Programme, section 5. English translation available in Geometric Galois Actions I (edited by L. Schneps and P. Lochak), LMS Lecture Notes Ser. 242, CUP 1997.
wikipedia o-minimal theory
Alessandro Berarducci, Definable groups in o-minimal structures, pdf; Cohomology of groups in o-minimal structures: acyclicity of the infinitesimal subgroup, J. Symbolic Logic 74:3 (2009), 891-900, MR2548466, euclid, doi, O-minimal spectra, infinitesimal subgroups and cohomology, J. Symbolic Logic 72 (2007), no. 4, pp. 1177–1193, MR2371198, euclid, doi
M. Edmundo, G. O. Jones, N. J. Peatfield, Sheaf cohomology in o-minimal structures, J. Math. Logic 6 (2006), no. 2, pp. 163–179, MR2317425, doi
Mario J. Edmundo, Luca Prelli, Invariance of o-minimal cohomology with definably compact supports, arxiv/1205.6124
Olivier Le Gal, Jean-Philippe Rolin, An o-minimal structure which does not admit $C^{\infty }$ cellular decomposition, Annales de l’institut Fourier 59:2 (2009), p. 543-562, MR2521427 Zbl 1193.03065 numdam