We study affine Deligne-Lusztig varieties in the affine flag manifold of an algebraic group, and in particular the question, which affine Deligne-Lusztig varieties are non-empty. Under mild assumptions on the group, we provide a complete answer to this question in terms of the underlying affine root system. In particular, this proves the corresponding conjecture for split groups stated in \cite{GHKR2}. The question of non-emptiness of affine Deligne-Lusztig varieties is closely related to the relationship between certain natural stratifications of moduli spaces of abelian varieties in positive characteristic.