The axioms of a category ensure that every finite number of composable morphisms has a (unique) composite.
Transfinite composition is a means to talk about morphisms in a category that behave as if they were the result of composing infinitely many morphisms.
Transfinite composition is indexed by ordinals. For convenience we first recall the definition of these assuming excluded middle in the ambient set theory (for definitions not assuming this see at ordinal and pointers given there):
A partial order is a set $S$ equipped with a relation $\leq$ such that for all elements $a,b,c \in S$
1) (reflexivity) $a \leq a$;
2) (transitivity) if $a \leq b$ and $b \leq c$ then $a \leq c$;
3) (antisymmetry) if a $a\leq b$ and $\b \leq a$ then $a = b$.
This we may and will equivalently think of as a category with objects the elements of $S$ and a unique morphism $a \to b$ precisely if $a\leq b$. In particular an order-preserving function between partially ordered sets is equivalently a functor between their corresponding categories.
A bottom element $\bot$ in a partial order is one such that $\bot \leq a$ for all a. A top element $\top$ is one for wich $a \leq \top$.
A partial order is a total order if in addition
4) (totality) either $a\leq b$ or $b \leq a$.
A total order is a well order if in addition
5) (well-foundedness) every non-empty subset has a least element.
An ordinal is the equivalence class of a well-order.
The successor of an ordinal is the class of the well-order with a top element freely adjoined.
A limit ordinal is one that is not a successor.
The finite ordinals are labeled by $n \in \mathbb{N}$, corresponding to the well-orders $\{0 \leq 1 \leq 2 \cdots \leq n-1\}$. Here $(n+1)$ is the successor of $n$. The first non-empty limit ordinal is $\omega = [(\mathbb{N}, \leq)]$.
Let $\mathcal{C}$ be a category, and let $I \subset Mor(\mathcal{C})$ be a class of its morphisms.
For $\alpha$ an ordinal (regarded as a category), an $\alpha$-indexed transfinite sequence of elements in $I$ is a diagram
such that
$X_\bullet$ takes all successor morphisms $\beta \stackrel{\leq}{\to} \beta + 1$ in $\alpha$ to elements in $I$
$X_\bullet$ is continuous in that for every nonzero limit ordinal $\beta \lt \alpha$, $X_\bullet$ restricted to the full-subdiagram $\{\gamma \;|\; \gamma \leq \beta\}$ is a colimiting cocone in $\mathcal{C}$ for $X_\bullet$ restricted to $\{\gamma \;|\; \gamma \lt \beta\}$:
The corresponding transfinite composition is the induced morphism
into the colimit of the diagram, schematically:
For purposes of constructive mathematics, the continuity condition should be stated as follows:
This actually includes $F(0) = X$ as a special case but says nothing when $\beta$ is a successor (so the successor clause is still required).
Transfinite composition plays a role in
For instance