The deployment of manipulators enhances the versatility and flexibility of unmanned aerial vehicles (UAVs) in aerial physical interaction tasks but also challenges their designs and controls due to variations in the center of gravity (CoG), moment of inertia, and reaction wrenches. This work presents a novel design of a two-degree-of-freedom dual-tool manipulator with invariant-center-of-gravity (ICoG) property. The ICoG conditions are strictly deduced, and a practical optimization-based parameter tuning method is proposed. A novel adaptive extended state observer (AESO)-based impedance control method is developed with actuator dynamics taken into account. The AESO can estimate and compensate for both the lumped disturbance, including the influences of moment-of-inertia variation and counter torque, and the unmeasurable states for the controller. In addition, a switching adaptive law is proposed to attenuate the peaking phenomenon under high observer gains. The impedance controller is designed using an auxiliary impedance tracking error to overcome the difficulty of the increased system order. The Lyapunov approach is used to evaluate the stability of the entire system. The proposed approach is implemented on a fully actuated hexarotor with a prototype of the ICoG manipulator. Comparative experiments are conducted to validate the effectiveness and advantages of the proposed design and control methods.