We propose a simple fast spectral method for the Boltzmann collision operator with general collision kernels. In contrast to the direct spectral method [L. Pareschi and G. Russo, SIAM J. Numer. Anal., 37 (2000), pp. 1217–1245; I. M. Gamba and S. H. Tharkabhushanam, J. Comput. Phys., 228 (2009), pp. 2012–2036], which requires O(N6) memory to store precomputed weights and has O(N6) numerical complexity, the new method has complexity O(MN4 log N), where N is the number of discretization points in each of the three velocity dimensions and M is the total number of discretization points on the sphere and M ≪ N2. Furthermore, it requires no precomputation for the variable hard sphere model and only O(MN4) memory to store precomputed functions for more general collision kernels. Although a faster spectral method is available [C. Mouhot and L. Pareschi, Math. Comp., 75 (2006), pp. 1833–1852] (with complexity O(MN3 logN)), it works only for hard sphere molecules, thus limiting its use for practical problems. Our new method, on the other hand, can apply to arbitrary collision kernels. A series of numerical tests is performed to illustrate the efficiency and accuracy of the proposed method.