We study the rigidity results for self-shrinkers in Euclidean space by restriction of the image under the Gauss map. The geometric properties of the target manifolds carry into effect. In the self-shrinking hypersurface situation Theorem 3.1and Theorem 3.2 not only improve the previous results, but also are optimal. In higher codimensional case, using geometric properties of the Grassmannian manifolds (the target manifolds of the Gauss map) we give a rigidity theorem for self-shrinking graphs.