The analysis of the vestibular system (VS) is an important research topic in medical image analysis. VS is a sensory structure in the inner ear for the perception of spatial orientation. It is believed several diseases, such as the Adolescent Idiopathic Scoliosis (AIS), are due to the impairment of the VS function. The morphology of the VS is thus of great research significance. A major challenge is that the VS is a genus-3 surface. The high-genus topology of the VS poses great challenges to find accurate pointwise correspondences between the surfaces and whereby perform accurate shape analysis. In this paper, we present a method to obtain the landmark constrained diffeomorphic registration between the VS surfaces based on the quasi-conformal theory. Given a set of corresponding landmarks on the VS surfaces, a diffeomorphism between the VS surfaces that matches the features consistently can be obtained. The basic idea is to iteratively search for an admissible Beltrami coefficient, which is associated to our desired landmark matching registration. With the obtained surface registrations, vertex-wise morphometric analysis can be carried out. Two types of geometric features are used for shape comparison. One is the collection of homotopic loops on each canals of the VS, which can be used to measure the local thickness of the canals. From the homotopic loops, centerlines can be extracted. By examining the deviations of the centerlines from the best fit planes, bendings of the canals can be detected. The second geometric feature is the minimal surface enclosed by the homotopic loop. From the minimal surfaces of each homotopic loops, cross-sectional area of the canals can be evaluated. To study the local shape difference more comprehensively, a complete shape index, which is defined using the Beltrami coefficients and surface curvatures, is used. We test proposed registration method on 15 VS of normal control subjects and 12 VS of patients suffering from AIS. Experimental results show the efficacy and accuracy of the proposed algorithm to compute the VS surface registration. Shape analysis has also been carried out using the proposed geometric features and shape index, which reveals shape differences in the posterior canal between normal and diseased AIS groups.