equivalences in/of $(\infty,1)$-categories
The notion of Segal category is one of the models for that of (∞,1)-category, given by regarding an $(\infty,1)$-category as an ∞Grpd-enriched (∞,1)-category.
So the notion can be understood as modelling the notion of an sSet-enrichment up to coherent homotopy, i.e. a weak enrichment. As such it is closely related to the notion of complete Segal space, which models the notion of an internal category in sSet.
Indeed, Segal categories may be considered with enrichment not just over sSet, but over other suitable model categories. In particular, an iterated enrichment over itself gives rise to the notion of Segal n-category which is a model for (∞,n)-categories.
Since the major difference between (small) $\mathcal{V}$-enriched categories and $\mathcal{V}$-internal categories is that in the first case the objects (as opposed to all the hom objects) form an ordinary set, while in the second these form an object of $\mathcal{V}$, too, accordingly a the definition of Segal category is like that of (complete) Segal space, only that the simplicial set of objects is required to be an ordinary set (a discrete simplicial set).
A Segal category is
a simplicial simplicial set ($\simeq$ bisimplicial set) $X \in [\Delta^{op}, sSet]$,
where we call $X_0$ the simplicial set of objects, $X_1$ the simplicial set of morphisms; and $X_k$ for $k \geq 2$ the simplicial set of sequences of composable morphisms of length $k$;
such that $X_0$ is a discrete (= constant) simplicial set;
and such that the Segal maps
induced by the spine inclusions $Sp[k] \hookrightarrow \Delta[k]$ are weak equivalences of simplicial sets for $k \geq 2$.
There is no condition that a Segal category be fibrant with respect to the Reedy model structure on bisimplicial sets.
For $X$ a Segal category, the fiber product simplicial set $X_1 \times_{X_0} X_1$ is manifestly the space of pairs of composable 1-morphisms in $X$, and the weak equivalence
given by the above definition together with the remaining face map $d_1 : X_2 \to X_1$ constitutes an ∞-anafunctor
given by the span
This encodes the composition operation in the Segal category $X$.
Accordingly, the analogous spans out of $X_k$ for $k \geq 3$ encode the associativity of this composition as well as all its coherences.
The category of bisimplicial sets carries a model category structure whose fibrant objects are the Reedy fibrant Segal categories. This model structure for Segal categories is a presentation of the (∞,1)-category of (∞,1)-categories.
The operadic generalization of Segal category is that of Segal operad. Segal categories are precisely those Segal operads whose only inhabited operations-spaces are those of unary operations.
Let $C$ be an ordinary small category and write $N(C) \in sSet$ for its nerve. Regard this as a bisimplicial set under the inclusion $sSet \simeq [\Delta^{op}, Set] \hookrightarrow [\Delta^{op}, sSet]$.
Then $N(C)$ is a Segal category. Each simplicial set $N(C)_k$ is discrete, for all $k \in \mathbb{N}$, and all the morphisms
are in fact isomorphisms / bijections of sets. This property of the nerve of an ordinary category goes by the name Segal condition and is what gave Segal categories its name.
One may also form the $n$-fold comma object-fiber product of a choice of base points $\pi_0(C) \to C$ with itself. This yields a Segal category incarnation of $C$ where in degree 1 we have the groupoid core of the arrow category of $C$. For more on this see at Segal space – Examples - From a category.
The idea of Segal categories goes back (implicitly) to
They were named Segal categories in
An overview is on pages 164 to 169 of
A discussion with emphasis on the comparison of the various model category structures is in
The generalization to Segal n-categories is discussed in section 2 of
In the more general context of enriched (∞,1)-categories, this is discussed in
and in section 2 of