We study convolution groups generated by completely monotone sequences and completely monotone functions. Using a convolution group, we define a fractional calculus for a certain class of distributions. When acting on causal functions, this definition agrees with the traditional Riemann-Liouville definition for t>0 but includes some singularities at t=0 so that the group property holds. Using this group, we are able to extend the definition of Caputo derivatives of order in (0,1) to a certain class of locally integrable functions without using the first derivative. The group property allows us to de-convolve the fractional differential equations to integral equations with completely monotone kernels, which then enables us to prove the general Gronwall inequality (or comparison principle) with the most general conditions. This then opens the door of a priori energy estimates of fractional PDEs. Some other fundamental results for fractional ODEs are also established within this frame under very weak conditions. Besides, we also obtain some interesting results about completely monotone sequences.