An operator topology is an abbreviation of a topology on a space of (continuous linear) operators between topological vector spaces over a fixed field of reals or complexes (possibly also p-adics, skewfield of quaternions etc.). In other words the hom-sets in the category of topological vector spaces as objects and continuous linear operators as morphisms are equipped with an operator topology.
There are many widely used topologies, some with standard names. Let be the set of continuous linear operators.
weak operator topology on is given by the basis of open neighborhoods of zero given by sets of the form where and . A sequence converges to is weak operator topology iff the sequence converges to in the weak topology on . We write or .
strong operator topology: the basis of neighborhoods of zero are given by sets , where and is a neighborhood of zero in . For convergence of sequences, we write or .
uniform operator topology: here we assume that are normed spaces with norms , . Then has a uniform operator topology induced by the norm given by the formula
The reason that in the definition of a unitary representation, the strong operator topology on is used and not the norm topology, is that only few homomorphisms turn out to be continuous in the norm topology.
and this will generally not be continuous in the norm topology, but is always continuous in the strong topology.