A linear and nonlinear stability analysis of a viscoelastic fluid in a porous medium layer saturated by a nanofluid with thermal conductivity and viscosity dependent on the nanoparticle volume fraction is studied. To simulate the momentum equation in porous media, a modified Darcy model has been used. To describe the rheological behavior of viscoelastic nanofluids, an Oldroyd-B type constitutive equation has been used. The onset criterion for stationary and oscillatory convection is derived analytically. The nonlinear theory based on the truncated representation of Fourier series method is used to find the transient heat and mass transfer.