This paper addresses two issues of image-based modeling for synthesizing photorealistic appearances of real objects under natural illumination conditions: capturing real-world illumination and modeling complex appearances of real objects for variable illumination. The appearance of an object changes significantly depending on its surface reflectance properties and the lighting condition of the scene in which it is placed. It is thus important to provide not only appropriate surface reflectance properties of an object in a scene but also natural illumination conditions of that scene so that a realistic appearance of the object can be synthesized under the provided natural illumination conditions. We first discuss an image-based approach and an inverse lighting approach for capturing and modeling real-world illumination. Then we shift our attention to how to model and synthesize complex appearances of real objects seen under natural illumination conditions.