If is a topological group, a topological space and the action continuous, then one can distinguish closed orbits from those which are not. Even when one starts with Hausdorff, the space of orbits is typically non-Hausdorff. (This problem is one of the motivations of the noncommutative geometry of Connes’ school.)
The orbit method is a method in representation theory introduced by Kirillov, Kostant and Souriau; it is a special case of geometric quantization. The orbit method is based on the study of the representations constructed from the coadjoint orbits with Kirillov symplectic structure. The terminology ‘geometric quantization’ allows for more general underlying spaces.
Given a compact Lie group with complexification and a unitary representation of on a finite-dimensional complex space , the real orbits of the highest weight vector agrees with the complex orbits, i.e. the orbits of the extension of this representation to the representation of the complexification. These are the coherent state orbits; there is also an infinite-dimensional version for reductive groups and representations which allows them (so-called coherent state representations).