Serre functors were introduced by Aleksei Bondal and Mikhail Kapranov to study admissible subcategories? of triangulated categories. A Serre functor on a triangulated category is an exact functor such that for any objects and , . It does not always exist, but when it does it is unique up to graded natural isomorphism.
In the original paper, the following definition was given.
Let be a -linear triangulated category with finite-dimensional Hom’s and algebraically closed. A Serre functor is an additive equivalence that commutes with the translation functor, with bi-functorial isomorphisms for any objects and , such that the composite
coincides with the isomorphism induced by .
In fact, the last commutativity condition can be deduced from just the bi-functoriality of , and commutativity with the translation functor also follows from a proposition below. Hence, the following definition is seen in later papers.
Let be a -linear category with finite-dimensional Hom’s and an arbitrary field. A Serre functor is an additive equivalence with bi-functorial isomorphisms for any objects and .
Of course, formally the definition could be used in categories enriched over a symmetric monoidal category with a sufficiently nice involution.
In the derived category of coherent sheaves on a smooth projective variety , the functor is a Serre functor, in view of Serre-Grothendieck duality?, where is the canonical sheaf and is the dimension of .
Any autoequivalence commutes with a Serre functor: there is a natural graded isomorphism of functors .
Any two Serre functors are connected by a canonical graded functorial isomorphism that commutes with the isomorphisms in the definition of the Serre functor.
The original paper and English translation:
А. И. Бондал, М. М. Капранов, Представимые функторы, функторы Серра и перестройки, Изв. АН СССР. Сер. матем., 53:6 (1989), 1183–1205 pdf
The following paper gives the corrected definition and also demonstrates the utility of the Serre functor as a tool for working with the derived category of coherent sheaves on a variety (c.f. Bondal-Orlov reconstruction theorem):