Just as an (orthogonal/unique) factorization system $(E,M)$ on a category $C$ gives a way to factor every morphism of $C$ as an $E$-map followed by an $M$-map, a ternary (orthogonal) factorization system $(E,F,M)$ gives a way to factor every map of $C$ as an $E$-map followed by an $F$-map followed by an $M$-map.
This is a special case of a notion of k-ary factorization system.
It turns out that a convenient way to state the definition is in terms of a pair of ordinary (orthogonal) factorization systems. We define a ternary factorization system on $C$ to consist of a pair $(L_1,R_1)$ and $(L_2,R_2)$ of ordinary orthogonal factorization systems such that $L_1 \subseteq L_2$ (or equivalently $R_2 \subseteq R_1$).
The three classes of map $(E,F,M)$ are then defined by $E=L_1$, $F = L_2\cap R_1$, and $M=R_2$. This is justified by:
Given a ternary factorization system as above, any morphism $f$ factors as
in an essentially unique way.
Consider the two ternary factorizations of $f$ obtained by
Note that both start with an $L_1$ map and end with an $R_2$ map. By a straightforward exercise in orthogonality, we can get comparison maps in both directions between these two factorizations which make them isomorphic. Therefore, since the first produces a middle map which is in $L_2$ and the second produces a middle map which is in $R_1$, this middle map must in fact be in $L_2\cap R_1$. Finally, any other such ternary factorization of $f$ induces an $(L_1,R_1)$ and $(L_2,R_2)$ factorization by composing pairwise, and uniqueness of these two implies uniqueness of the ternary factorization.
More explicitly, if we factor $f$ as
with $\lambda_i, l_i\in L_i, \rho_j,r_j\in R_j$, then we obtain from the lifting problems
arrows $s,t,a_1,a_2$: now it’s easy to see that $s,t\in L_2\cap R_1$, and that $a_i\in L_i\cap R_i= Iso$.
Conversely, just as for a binary factorization system, the extra requirement of orthogonality can be deduced from uniqueness of the factorizations, a unique and functorial ternary factorization implies that it “splits” into a pair of binary factorization systems, i.e. a ternary factorization system as defined here. This is remarked on here.
One can also characterize the notion in terms of a ternary factorization with a “ternary orthogonality” property; see the paper of Pultr and Tholen referenced below.
In addition to $L_1$, $R_1$, $L_2$, $R_2$, and $L_2\cap R_1$, a ternary factorization system also determines a sixth important class of morphisms, namely those whose $(L_2\cap R_1)$-part is an isomorphism, or equivalently those that can be factored as an $L_1$-map followed by an $R_2$-map. We therefore call this class $R_2 L_1$.
In a ternary factorization system, $L_1 = L_2 \cap R_2L_1$ and $R_2 = R_1 \cap R_2L_1$.
In both cases $\subseteq$ is obvious. Conversely, if $f \in L_2 \cap R_2 L_1$, say $f = m e$ for $m\in R_2$ and $e\in L_1$, then orthogonality in the square
exhibits $f$ as a retract of $e$ in $Arr(C)$, whence $f\in L_1$ since $L_1$ is closed under retracts.
In Top, let $L_1=$ quotient maps, $R_1=$ injective continuous maps, $L_2=$ surjective continuous functions, and $R_2=$ subspace embeddings. Here $L_2\cap R_1=$ bijective continuous maps, and the two intermediate objects in the ternary factorization of a continuous map are obtained by imposing the coarsest and the finest compatible topologies on its set-theoretic image.
More generally, if a category has both (epi, strong mono) and (strong epi, mono) factorizations, then since strong epis are epi, we have a ternary factorization. Here $L_2\cap R_1$ is the class of monic epics, sometimes called bimorphisms. The maps in $R_2 L_1$ are sometimes called strict morphisms.
On Cat there is a 2-categorical version of a ternary factorization system, determined by the 2-categorical factorization systems (eso+full, faithful) and (eso, full and faithful). Here $L_2\cap R_1$ is the class of eso+faithful functors, while $R_2 L_1$ is the class of full functors. This factorization system plays an important role in the study of stuff, structure, property.
Restricted to groupoids this is the 1-image-2-image factorization, the 3-stage Postnikov system of groupoids.
On Topos there is also a 2-categorical ternary factorization system composed of the binary 2-categorical factorization systems (hyperconnected, localic) and (surjection, inclusion). Here the maps in $L_2\cap R_1$ have no name other than “localic surjections,” and those in $R_2 L_1$ have no established name (although they are briefly mentioned in A4.6.10 of the Elephant).
Suppose that $C$ has a binary factorization system $(E,M)$ and that $p\colon A\to C$ is an ambifibration? relative to $(E,M)$: i.e. every arrow in $E$ has an opcartesian lift and every arrow in $M$ has a cartesian lift. (In particular, $p$ could be a bifibration.) Then there is a ternary factorization system on $A$ for which $L_1$ is the class of opcartesian arrows over $E$, $R_2$ is the class of cartesian arrows over $M$, and $L_2\cap R_1$ is the class of vertical arrows (those lying over identities). See this comment.
For instance, the above factorization system on $Top$ is induced in this way via the forgetful functor $Top\to Set$ from the (epi,mono) factorization system on Set.
A similar example is given by a span $A \overset{p}{\leftarrow} E \overset{q}{\to} B$ of categories where $p$ is a fibration whose cartesian morphisms are $q$-vertical and $q$ is an opfibration? whose opcartesian morphisms are $p$-vertical (that is, the span $(p,q)$ is both a left and a right fibration in the sense of Street). Then the two factorization systems on $E$ given by the $q$-opcartesian and $q$-vertical morphisms on the one hand, and the $p$-vertical and $p$-cartesian morphisms on the other, satisfy the $L_1 \subseteq L_2$ condition above, so that every morphism in $E$ factors as a $q$-opcartesian morphism followed by a morphism that is both $p$- and $q$-vertical, followed by a $p$-cartesian morphism.
Such a span is a two-sided fibration if $L_1R_2 \subseteq R_2L_1$, that is if the three-way factorization of the composite of a $p$-cartesian morphism followed by a $q$-opcartesian one has its middle term an isomorphism.
The notion of model category involves a pair of weak factorization systems called (acyclic cofibration, fibration) and (cofibration, acyclic fibration) which are compatible in the same sense as above. However, non-uniqueness of these factorizations means that the resulting “ternary factorization” of a morphism is not unique. The class corresponding to $R_2 L_1$ is important, however: it is precisely the class of weak equivalences.
The notion of k-ary factorization system is a generalization to factorizations into $k$ morphisms.
Just as strict factorization systems can be identified with distributive laws in the bicategory of spans, so “strict” ternary (and k-ary) factorization systems can be identified with iterated distributive laws in $Span$.
A. Pultr and W. Tholen, Free Quillen Factorization Systems. Georgian Math. J.9 (2002), No. 4, 807-820