We consider closed orientable hypersurfaces in a wide class of warped product manifolds
which include space forms, deSitter-Schwarzschild and Reissner-Nordstr\"{o}m manifolds. By using a new integral formula or Brendle's Heintze-Karcher type inequality,
we present some new characterizations of umbilic hypersurfaces.
These results can be viewed as generalizations of the classical Jellet-Liebmann theorem and the Alexandrov theorem in Euclidean space. In particular, Corollary 1.8 implies that the embeddedness condition in Theorem 2 of [Brendle-Eichmair, JDG. 2013] is not necessary.