A Lie groupoid is a groupoid internal to smooth manifolds. This is a joint generalization of smooth manifolds and Lie groups to higher differential geometry.
Regarded in the more general context of smooth groupoids/smooth stacks, Lie groupoids present certain well-behaved such objects, often called differentiable stacks.
A Lie groupoid $X:=X_1 \rightrightarrows X_0$ is a groupoid such that both the space of arrows $X_1$ and the space of objects $X_0$ are smooth manifolds, all structure maps are smooth, and source and target maps $s, t: X_1\rightrightarrows X_0$ are surjective submersions.
A Lie groupoid $X$ is an internal groupoid in the category Diff of smooth manifolds.
Since Diff does not have all pullbacks, to ensure that this definition makes sense, one needs to ensure that the space $X_1 \times_{s,t} X_1$ of composable morphisms is an object of Diff. This is achieved either by adopting the definition of internal groupoid in the sense of Ehresmann, which includes as data the smooth manifold of composable pairs, or by taking the conventional route and demanding that the source and target maps $s,t : X_0 \to X_1$ are submersions. This ensures the pullback exists to define said manifold or composable pairs. Therefore a definition used in most modern differential geometry literature is as we see above.
But for most practical purposes, the apparently evident 2-category $Grpd(Diff)$ of such internal groupoids, internal functors and internal natural transformations is not the correct 2-category to consider. One way to see this is that the axiom of choice fails in Diff, which means that an internal functor $X \to Y$ of internal groupoids which is essentially surjective and full and faithful may nevertheless not be an equivalence, in that it may not have a weak inverse in $Grpd(Diff)$.
See the section 2-Category of Lie groupoids below.
A bit more general than a Lie groupoid is a diffeological groupoid.
Originally Lie groupoids were called (by Ehresmann) differentiable groupoids (and also one considered differentiable categories). Sometime in the 1980s there was a change of terminology to Lie groupoid and differentiable stacks. (reference?)
One definition which Ehresmann introduced in his paper Catégories topologiques et catégories différentiables (see below) is that of locally trivial groupoid. It is defined more generally for topological categories, and extends in an obvious way to topological groupoids, and Lie categories and groupoids. For a topological (resp. Lie) category $X$, let $X_1^{iso}$ denote the subspace (resp. submanifold) of invertible arrows . (This always exists, by general abstract nonsense - I should look up the reference, it’s in Bunge-Pare I think - DR)
A topological groupoid $X_1 \rightrightarrows X_0$ is locally trivial if for every point $p\in X_0$ there is a neighbourhood $U$ of $p$ and a lift of the inclusion $\{p\} \times U \hookrightarrow X_0 \times X_0$ through $(s,t):X_1^{iso}\to X_0 \times X_0$.
Clearly for a Lie groupoid $X_1^{iso} = X_1$. It is simple to show from the definition that for a transitive Lie groupoid, $(s,t)$ has local sections. Ehresmann goes on to show a link between smooth principal bundles and transitive, locally trivial Lie groupoids. See locally trivial category for details.
As usual for internal categories, the naive 2-category of internal groupoids, internal functors and internal natural transformations is not quite “correct”. One sign of this is that the axiom of choice fails in Diff so that an internal functor which is an essentially surjective functor and a full and faithful functor may still not have an internal weak inverse.
One way to deal with this is to equip the 2-category with some structure of a homotopical category and allow morphisms of Lie groupoids to be anafunctors, i.e. spans of internal functors $X \stackrel{\simeq}{\leftarrow} \hat X \to Y$.
Such generalized morphisms – called Morita morphisms or generalized morphisms in the literature – are sometimes modeled as bibundles and then called Hilsum-Skandalis morphisms.
Another equivalent approach is to embed Lie groupoids into the context of 2-topos theory:
The (2,1)-topos $Sh_{(2,1)}(Diff)$ of stacks/2-sheaves on Diff may be understood as a nice 2-category of general groupoids modeled on smooth manifolds. The degreewise Yoneda embedding allows to emebed groupoids internal to $Diff$ into stacks on $Diff$. this wider context contains for instance also diffeological groupoids.
Regarded inside this wider context, Lie groupoids are identified with differentiable stacks. The (2,1)-category of Lie groupoids is then the full sub-$(2,1)$-category of $Sh_{(2,1)}(Diff)$ on differentiable stacks.
For more comments on this, see also the beginning of ∞-Lie groupoid.
As the infinitesimally approximation to a Lie group is a Lie algebra, so the infinitesimal approximation to a Lie groupoid is a Lie algebroid.
See
Every smooth manifold $X$ is a 0-truncated Lie groupoid.
For every Lie group $G$ the one-object delooping groupoid $\mathbf{B}G$ is a Lie groupoid.
The Lie group $G$ itself is a 0-truncated group object in the 2-category or Lie groupoids.
Every Lie 2-group is in particular a Lie groupoid: a group object in the category of Lie groupoids.
The inner automorphism 2-group $\mathbf{E}G = INN(G) = G//G$ is a Lie groupoid. There is an obvious morphism $\mathbf{E}G \to \mathbf{B}G$.
For every $G$-principal bundle $P \to X$ there is its Atiyah Lie groupoid $At(P)$.
The fundamental groupoid $\Pi_1(X)$ of a smooth manifold is naturally a Lie groupoid.
The path groupoid of a smooth manifold is naturally a diffeological groupoid.
The Cech groupoid $C(U)$ of a cover $\{U_i \to X\}$ of a smooth manifold is a Lie groupoid.
Every foliation gives rise to its holonomy groupoid.
An orbifold is a Lie groupoid.
An anafunctor $X \stackrel{\simeq}{\leftarrow} C(U) \to \mathbf{B}G$ from a smooth manifold $X$ to $\mathbf{B}G$ is a Cech cocycle in degree 1 with values in $G$, classifying $G$-principal bundle $P$.
The (1-categorical) pullback
is a Lie groupoid equivalent to this principal bundle $P$.
(For more on the general phenomenon of which this is a special case see principal ∞-bundle and universal principal ∞-bundle.)
Similarly an anafunctor from $P_1(X)$ to $\mathbf{B}G$ is a connection on a bundle (see there for details).
Topological and differentiable (or smooth, “Lie”) groupoids (and more generally categories) were introduced in
Reviews and developments of the theory of Lie groupoids include
Pradines, ….
Kirill Mackenzie, General Theory of Lie Groupoids and Lie Algebroids, Cambridge University Press, 2005, xxxviii + 501 pages (website)
Kirill Mackenzie, Lie groupoids and Lie algebroids in differential geometry, London Mathematical Society Lecture Note Series, 124. Cambridge University Press, Cambridge, 1987. xvi+327 pp (MathSciNet)
Discussion in the context of foliation theory (foliation groupoids) is in
The relation to differentiable stacks is discussed/reviewed in section 2 of
Lie groupoids as a source for groupoid convolution C*-algebras are discussed in
Expository discussion of various kinds of groupoids is also in