Instability of natural convection in nanofluids is investigated in this work. As a result of Brownian motion and thermophoresis of nanoparticles, the critical Rayleigh number is shown to be much lower, by one to two orders of magnitude, as compared to that for regular fluids. The highly promoted turbulence, in the presence of nanoparticles for as little as 1% in volume fraction, significantly enhances heat transfer in nanofluids, which may be much more pronounced than the enhancement of the effective thermal conductivity alone. Seven dominating groups are extracted from the nondimensional analysis. By extending the method of eigenfunction expansions in conjunction with the method of weighted residuals, closed-form solutions are derived for the Rayleigh number to justify such remarkable change by the nanoparticles at the onset of instability.