The effect of reflection and transparency, which results from shiny or glass-like transparent materials, is superimposed on captured images of many actual outdoor scenes. The presence of such an incidental effect in a captured image has made it difficult to apply computer vision algorithms and has led to erroneous results. Moreover, it disturbs the texture acquisition of an outdoor scene, an important topic for the Computer Graphics (CG) and Inteligence Transportation System (ITS) community. This paper presents an optimal method for the automatic separation of reflected and transparent layers even if the scene is complicated with view-dependent effects and depth disparity in a 3D environment. The method is based on epipolar plane image (EPI) analysis. The method is not like the conventional edge-based EPI analysis, but instead it is a color-based analysis. First, we separate EPI into two layers by our original color-based EPI analysis. Then, original image sequence is separated into reflected and transparent layers by using the separated EPI. To demonstrate the effectiveness of our method, we implement the algorithm and present the results of experiments using synthesized and real scene images including indoor and outdoor scenes.