A hereditarily finite set is a finite set of hereditarily finite sets; this circular definition is usually interpreted recursively, although you can also interpret it corecursively to get ill-founded hereditarily finite sets. The set of all (well-founded) hereditarily finite sets (which is infinite, and not hereditarily finite itself) is written to show its place in the von Neumann hierarchy of pure sets.
As a property of a set, being hereditarily finite is equivalent (up to isomorphism of sets) to simply being finite. So the ‘hereditary’ part is meaningful only in material set theory, not structurally, unless you see it as a property of a pure set represented structurally as a membership tree.
There are countably many hereditarily finite sets, up to equality (in material set theory), and in fact they can be neatly enumerated as follows: Given a natural number , write in base ; the th hereditarily finite set is a member of the th one if the th digit of is . (This definition is well-founded, because the highest non-zero digit of must have position at most , which is less than .)
So the hereditarily finite sets are as follows:
In this table, we've indicated the representations of and in the most common models of natural numbers as pure sets, those of Zermelo (where ) and of von Neumann (where ); these both begin with and but diverge thereafter. (Von Neumann's representation is favoured now, as it allows each natural number to have itself as its cardinal number, a situation that generalises to infinite limit ordinal numbers.) However, the existence of this enumeration shows that another representation of natural numbers as pure sets is to use all hereditarily finite sets.
The set of hereditarily finite sets is a Grothendieck universe (unless you phrase the definition specifically to rule this out). Thus the axiom of infinity (which guarantees the existence of some model of the set of natural numbers) can be seen as following from a very simple universe axiom: that some Grothendieck universe exists. Conversely, if any natural numbers object exists in the category of sets, then you can form the universe (using the axiom of replacement) by performing the above enumeration.
In constructive mathematics, one gets different notions of hereditarily finite set depending on exactly how one defines finite set. The enumeration above works if you use the strictest sense, but you need to close under taking subsets (or use subfinite sets to start with) to get a Grothendieck universe in material set theory.