Loosely speaking, the boundary of a subset of topological space consists of those points in that are neither ‘fully in’ nor are ‘fully not in’ .
Letting denote set-theoretic complementation, . It is a closed set. If we consider restricted to closed sets as an operation on closed sets, then it becomes a special case of the boundary operator on a co-Heyting algebra; see there for further properties.
In a manifold with boundary of dimension the boundary is the collection of points which do not have a neighborhood diffeomorphic to an open n-ball, but do have a neighborhood homeomorphic to a half-ball, that is, an open ball intersected with closed half-space.
One reason behind the notation may be this (cf. co-Heyting boundary):
Let be topological spaces. Then for closed subsets and , the Leibniz rule holds.
Notice the conclusion must fail if , are not closed, since in this case is not closed (it doesn’t include ).
The interior operation preserves intersections, so . Its complement is , whose intersection with is .
Since is open, we have
where the right side is a disjoint union of open sets. is connected, so or . The latter cannot occur since is inhabited. So ; by symmetry .
The Leibniz rule shows that the boundary operator is better behaved when restricted to the lattice of closed subsets. Since this lattice forms a co-Heyting algebra, one is led to study algebraic operators axiomatizing properties of (Zarycki 1927) on these, the so called co-Heyting boundary operators.
Since the lattice of subtoposes of a given topos carries a co-Heyting algebra structure, it becomes possible to define (co-Heyting) boundaries of subtoposes and thereby even boundaries of the geometric theories that the subtoposes correspond to! Intuitively, such a boundary of a theory consists of those geometric sequents that neither ‘fully follow’ from nor ‘fully contradict’ .
The interior of a subtopos of a Grothendieck topos is defined in an exercise of SGA4 as the largest open subtopos contained in . The boundary is then defined as the subtopos complementary to the (open) join of the exterior subtopos and in the lattice of subtoposes.
M. Zarycki, Quelque notions fondamentales de l’Analysis Situs au point de vue de l’Algèbre de la Logique , Fund. Math. IX (1927) pp.3-15. (pdf)