An $\Omega$-group is a group equipped with additional algebraic operations (of signature $\Omega$) that distribute over the group operations.
We take an algebraic structure in a traditional sense as a set $\Omega$ with a (not necessarily finite) number of operations $\alpha$ of various arity and satisfying some axioms (not necessarily of first order). That is, we are discussing objects of an equationally presentable or algebraic category.
An $\Omega$-group is an algebraic structure which amounts to a group (usually written additively but not necessarily commutative) together with a set $\Omega$ of operations of any arity, such that for each $n$-ary operation $\alpha \in \Omega$, distributivity holds in each variable over the group operations:
(This states only distributivity over addition; however, distributivity over all other group operations follows.)
The classical examples are of course groups (where $\Omega$ is empty), rngs (where $\Omega$ consists of only multiplication), and rings (where $\Omega$ consists of multiplication and the nullary operator that gives the multiplicative identity). Given a fixed ground ring $k$, the modules over $k$ form another example: each element of $k$ gives a unary multiplication operation.
The older term group with operators is traditionally used for $\Omega$-groups when only unary operations are considered (as in the case of modules).
The general theory of $\Omega$-groups is similar to the basics of group and ring theory, including normal subgroups / ideals / submodules, quotient? $\Omega$-groups, Noether’s isomorphism theorem?s, etc. For example, the Jordan–Holder theorem? holds: if there is a composition series, then every two composition series are equivalent up to permutation of factors. An obvious horizontal categorification of $\Omega$-groups is also interesting.
Note that $\Omega$ is a capital Greek letter; $\omega$-group is rather a synonym for (for some people strict) $\infty$-groupoid with a single object, hence nothing to do with $\Omega$-groups.
Wikipedia, Group with operators.
N. Bourbaki, Algebra I, ch. 1-3.
E. I. Khukhro, Local nilpotency in varieties of groups with operators, Russ. Acad. Sci. Sbornik Mat. 78 379, 1994. (doi)
Grace Orzech, Obstruction theory in algebraic categories I, II, J. Pure Appl. Algebra 2 (1972) 287-340, 315–340.