Let U be a connected non-singular quasi-projective variety and f : A → U a family of abelian varieties of dimension g. Suppose
that the induced map U → Ag is generically finite and there is a compactification Y with complement S = Y \U a normal crossing
divisor such that Ω1 Y (log S) is nef and ωY (S) is ample with respect to U.
We characterize whether U is a Shimura variety by numerical data attached to the variation of Hodge structures, rather than by
properties of the map U →Ag or by the existence of CM points.
More precisely, we show that f : A → U is a Kuga fibre space, if and only if two conditions hold. First, each irreducible local subsystem V of R1f∗CA is either unitary or satisfies the Arakelov equality. Second, for each factor M in the universal cover of U
whose tangent bundle behaves like that of a complex ball, an iterated Kodaira-Spencer map associated with V has minimal possible
length in the direction of M. If in addition f : A → U is rigid, it is a connected Shimura subvariety of Ag of Hodge type.