Ferromagnetic soft materials (FSM) can generate flexible movement and shift morphology in response to an external magnetic field. They have been engineered to design products in a variety of promising applications, such as soft robots, compliant actuators, or bionic devices, et al. By using different patterns of magnetization in the soft elastomer matrix, ferromagnetic soft matters can achieve various shape changes. Although many magnetic soft robots have been designed and fabricated, they are limited by the designers’ intuition. Topology optimization (TO) is a systematically mathematical method to create innovative structures by optimizing the material layout within a design domain without relying on the designers’ intuition. It can be utilized to architect ferromagnetic soft active structures. Since many of these ‘soft machines’ exist in the form of thin-shell structures, in this paper, the extended level set method (X-LSM) and conformal mapping theory are employed to carry out topology optimization of the ferromagnetic soft actuator on manifolds. The objective function consists of a sub-objective function for the kinematics requirement and a sub-objective function for minimum compliance. Shape sensitivity analysis is derived using the material time derivative and adjoint variable method. Two examples, including a circular shell actuator and a flytrap structure, are studied to demonstrate the effectiveness of the proposed framework.