The bifurcations of penetrative Rayleigh-Benard convection in cylindrical containers are studied by the linear stability analysis (LSA) combined with the direct numerical simulation (DNS) method. The working fluid is cold water near 4 degrees C, where the Prandtl number Pr is 11.57, and the aspect ratio (radius/height) of the cylinder ranges from 0.66 to 2. It is found that the critical Rayleigh number increases with the increase in the density inversion parameter (m). The relationship between the normalized critical Rayleigh number (Ra-c((m))/Ra-c(0)) and (m) is formulated, which is in good agreement with the stability results within a large range of (m). The aspect ratio has a minor effect on Ra-c((m))/Ra-c(0). The bifurcation processes based on the axisymmetric solutions are also investigated. The results show that the onset of axisymmetric convection occurs through a trans-critical bifurcation due to the top-bottom symmetry breaking of the present system. Moreover, two kinds of qualitatively different steady axisymmetric solutions are identified.