We study a two-dimensional charged particle interacting with a magnetic field, in general non-homogeneous, perpendicular to the plane, a confining potential, and a point interaction. If the latter moves adiabatically along a loop, the state corresponding to an isolated eigenvalue acquires a Berry phase. We derive an expression for it and evaluate it in several examples such as a homogeneous field, a magnetic whisker, a particle confined at a ring or in quantum dots, a parabolic and a zero-range one. We also discuss the behavior of the lowest Landau level in this setting obtaining an explicit example of the WilczekZee phase for an infinitely degenerated eigenvalue.