Magnetic devices such as actuators and sensors are composed of ferromagnetic materials, permanent magnets, and coils. Thus, in optimizations of magnetic problems, consideration of multi-materials is very important. This work presents a phase field-based design method that simultaneously considers iron cores and permanent magnets, where the time evolution of the phase fields is based on an Allen-Cahn equation. The proposed method is numerically implemented to design a C-core actuator, a Halbach magnet, and a magnetostrictive sensor. The design sensitivities are derived using the adjoint variable method, and the augmented Lagrangian is employed to deal with the volume constraints.