A series of numerical simulation of the flow and heat transfer in modeled counterflow heat exchangers with oblique wavy walls has been made toward optimal shape design of recuperators. The effects of oblique angles and amplitudes of the wavy walls are systematically evaluated, and the heat transfer and pressure loss characteristics are investigated. It is found that counter-rotating streamwise vortices are induced by the wavy walls, and the flow field has been drastically modified due to the intense secondary flow. By using the optimum set of oblique angle and wave amplitude, significant heat transfer enhancement has been achieved at the cost of relatively small pressure loss, and the $j\u2215f$ factor becomes much larger than that of straight square duct or conventional compact recuperators. When thermal coupling of hot and cold fluid passages is considered, the heat transfer is found to be strongly dependent on the arrangement of counterflow passages. The total heat transfer surface area required for a given pumping power and heat transfer rate can be reduced by more than 60% if compared to the straight square duct.