A group scheme is a group object in the category of schemes. Therefore it may be considered a generalization of an algebraic group. In the functor of points formalism, a group scheme over a scheme is a functor
(where Grp is the category of discrete groups) such that the composition with the forgetful functor is representable.
See also formal group.
M. Artin, J. E. Bertin, M. Demazure, P. Gabriel, A. Grothendieck, M. Raynaud, J.-P. Serre, Schemas en groupes, i.e. SGA III-1, III-2, III-3
M. Demazure, P. Gabriel, Groupes algebriques, tome 1 (later volumes never appeared), Mason and Cie, Paris 1970
W. Waterhouse, Introduction to affine group schemes, GTM 66, Springer 1979.
D. Mumford, Abelian varieties, 1970, 1985.
J. C. Jantzen, Representations of algebraic groups, Acad. Press 1987 (Pure and Appl. Math. vol 131); 2nd edition AMS Math. Surveys and Monog. 107 (2003; reprinted 2007)