This paper presents a unified haptic modeling method for physical systems irrespective of their types. Previously, we have proposed an energy-based approach which haptically renders rigid and deformable objects without reference to their material. However, it is not easy to represent the haptic behavior of a fluid or fluid-deformable object with the previously proposed method because there is no method for handling fluid objects and no additional interface between a fluid object and the other objects. Therefore, this paper proposes a modeling framework in which rigid, deformable, and fluid objects are represented by one modeling method.