Types of quantum field thories
Ordinary heterotic string theory is the study of the perturbation series of correlators of a certain 2-dimensional SCFT over surfaces. The resulting S-matrix is interpreted as encoding the scattering amplitudes of strings propagating in some target space.
By general electric-magnetic duality (see Freed for a formal treatment in differential cohomology) one expects there to be a dual theory theory where the -dimensional strings are replaced by their magnetic duals, which are -branes.
The study of the corresponding perturbation series over correlators of the 6-dimensional fivebrane worldvolume SCFT over 6-volumes is far from tractable, but a handful of consistency checks exist, that the corresponding dual heterotic string theory makes sense.
Where the quantum anomaly-cancellation for the heterotic string involves the demand for (twisted) string structures, that for the dual theory involves fivebrane structures (which gives these their name).
Review of the history of the dual heterotic string conjecture is in
The worldvolume fermion quantum anomaly of the super-fivebrane is discussed in
K. Lechner, M. Tonin, Worldvolume and target space anomalies in the D=10 super–fivebrane sigma–model (arXiv:hep-th/9603094)
J. A. Dixon, M. J. Duff, J. C. Plefka, Putting the string/fivebrane duality to the test (arXiv:hep-th/9208055)
On p. 3 and 4 the latter reviews the worldsheet description of the relevant terms for the string and then look at the corresponding situation for the 5-brane on p. 6.
So this is the EM-dual analog of the Killingback-Witten-computation of the fermionic anomaly that leads to string structures in the heterotic string (see there). Where the former involves cancelling the first fractional Pontryagin class, this involves the second.
For a description of the general mechanism in differential cohomology at work here, see