In this paper, a novel penalty method based on the immersed boundary formulation is proposed for simulating the transient Stokes flow with an inextensible interface enclosing a suspended solid particle. The main idea of this approach relies on the penalty techniques by modifying the constitutive equation of Stokes flow to weaken the incompressibility condition, relating the surface divergence to the elastic tension $\sigma$ to relax the interface's inextensibility, and connecting the particle surface-velocity with the particle surface force ${\mbf F}$ to regularize the particle's rigid motion. The advantage of these regularized governing equations is that when they are discretized by the standard centered difference scheme on a staggered grid, the resulting linear system can easily be reduced by eliminating the unknowns $p_h$, $\sigma_h$ and ${\mbf F}_h$ directly, so that we just need to solve a smaller linear system of the velocity approximation ${\mbf u}_h$. This advantage is preserved and even enhanced when such approach is applied to the transient Stokes flow with multiple compound vesicles. Moreover, this smaller linear system is symmetric and negative-definite, which enables us to use efficient linear solvers. Another important feature of the proposed method is that the discretization scheme is unconditionally stable in the sense that an appropriately defined energy functional associated with the discrete system is decreasing and hence bounded in time. We numerically test the accuracy and stability of the immersed boundary discretization scheme. The tank-treading and tumbling motions of inextensible interface with a suspended solid particle in the simple shear flow will be studied extensively. The simulation of the motion of multiple compound vesicles will be performed as well. Numerical results illustrate the superior performance of the proposed penalty method.