We study the stability of traveling waves of the nonlinear Schrödinger equation
with nonzero condition at infinity obtained via a constrained variational approach.
Two important physical models for this are the Gross–Pitaevskii (GP) equation
and the cubic-quintic equation. First, under a non-degeneracy condition we prove a
sharp instability criterion for 3D traveling waves of (GP), which had been conjectured
in the physical literature. This result is also extended for general nonlinearity
and higher dimensions, including 4D (GP) and 3D cubic-quintic equations. Second,
for cubic-quintic type nonlinearity, we construct slow traveling waves and prove
their nonlinear instability in any dimension. For dimension two, the non-degeneracy
condition is also proved for these slow traveling waves. For general traveling waves
without vortices (that is nonvanishing) and with general nonlinearity in any dimension,
we find a sharp condition for linear instability. Third, we prove that any 2D
traveling wave of (GP) is transversally unstable, and we find the sharp interval
of unstable transversal wave numbers. Near unstable traveling waves of all of the
above cases, we construct unstable and stable invariant manifolds.