We consider densities $D_\Sigma(A)$, $\overline{D}_\Sigma(A)$ and $\underline{D}_\Sigma(A)$ for a subset $A$ of $\mathbb{N}$ with respect to a sequence $\Sigma$ of finite subsets of $\mathbb{N}$ and study Fourier coefficients of ergodic, weakly mixing and strongly mixing $\times p$-invariant measures on the unit circle $\mathbb{T}$. Combining these, we prove the following measure rigidity results: on $\mathbb{T}$, the Lebesgue measure is the only non-atomic $\times p$-invariant measure satisfying one of the following: (1) $\mu$ is ergodic and there exist a F\o lner sequence $\Sigma$ in $\mathbb{N}$ and a nonzero integer $l$ such that $\mu$ is $\times (p^j+l)$-invariant for all $j$ in a subset $A$ of $\mathbb{N}$ with $D_\Sigma(A)=1$; (2) $\mu$ is weakly mixing and there exist a F\o lner sequence $\Sigma$ in $\mathbb{N}$ and a nonzero integer $l$ such that $\mu$ is $\times (p^j+l)$-invariant for all $j$ in a subset $A$ of $\mathbb{N}$ with $\overline{D}_\Sigma(A)>0$; (3) $\mu$ is strongly mixing and there exists a nonzero integer $l$ such that $\mu$ is $\times (p^j+l)$-invariant for infinitely many $j$. Moreover, a $\times p$-invariant measure satisfying (2) or (3) is either a Dirac measure or the Lebesgue measure.
As an application we prove that for every increasing function $\tau$ defined on positive integers with $\lim_{n\to\infty}\tau(n)=\infty$, there exists a multiplicative semigroup $S_\tau$ of $\mathbb{Z}^+$ containing $p$ such that $|S_\tau\cap[1,n]|\leq (\log_p n)^{\tau(n)}$ and the Lebesgue measure is the only non-atomic ergodic $\times p$-invariant measure which is $\times q$-invariant for all $q$ in $S_\tau$.