We construct natural virtual fundamental classes for nested Hilbert schemes on a nonsingular projective surface S. This allows us to define new invariants of S that recover some of the known important cases such as Poincare invariants of Durr-Kabanov-Okonek and the stable pair invariants of Kool-Thomas. In the case of the nested Hilbert scheme of points, we can express these invariants in terms of integrals over the products of Hilbert scheme of points on S, and relate them to the vertex operator formulas found by Carlsson-Okounkov. The virtual fundamental classes of the nested Hilbert schemes play a crucial role in the Donaldson-Thomas theory of local-surface-threefolds that we study in [GSY17b].
Let S be a projective simply connected complex surface and be a line bundle on S. We study the moduli space of stable compactly supported 2-dimensional sheaves on the total spaces of . The moduli space admits a ℂ∗-action induced by scaling the fibers of . We identify certain components of the fixed locus of the moduli space with the moduli space of torsion free sheaves and the nested Hilbert schemes on S. We define the localized Donaldson-Thomas invariants of by virtual localization in the case that twisted by the anti-canonical bundle of S admits a nonzero global section. When pg(S)>0, in combination with Mochizuki's formulas, we are able to express the localized DT invariants in terms of the invariants of the nested Hilbert schemes defined by the authors in [GSY17a], the Seiberg-Witten invariants of S, and the integrals over the products of Hilbert schemes of points on S. When is the canonical bundle of S, the Vafa-Witten invariants defined recently by Tanaka-Thomas, can be extracted from these localized DT invariants. VW invariants are expected to have modular properties as predicted by S-duality.
We discuss Donaldson-Thomas (DT) invariants of torsion sheaves with 2 dimensional support on a smooth projective surface in an ambient non-compact Calabi Yau fourfold given by the total space of a rank 2 bundle on the surface. We prove that in certain cases, when the rank 2 bundle is chosen appropriately, the universal truncated Atiyah class of these codimension 2 sheaves reduces to one, defined over the moduli space of such sheaves realized as torsion codimension 1 sheaves in a noncompact divisor (threefold) embedded in the ambient fourfold. Such reduction property of universal Atiyah class enables us to relate our fourfold DT theory to a reduced DT theory of a threefold and subsequently then to the moduli spaces of sheaves on the base surface using results in arXiv:1701.08899 and arXiv:1701.08902 and . We finally make predictions about modularity of such fourfold invariants when the base surface is an elliptic K3.