During video transmission, errors may happen normally. Addressing error resiliency, the reference modification (RM) approach at video compression encoder is used to modify the original reference to a modified one, which is less sensitive to errors. Apparently it is a bit rate overhead compared to original reference based conventional encoder. Therefore to reduce such bit rate overhead while still keep the error resilience performance is a big issue. A low bit rate overhead based RM approach is proposed in this paper, where the decoder information is exploited to perform encoding. In the proposal, the modified reference has more correlation to original reference, thus can reduce the bit rate overhead. In addition, the proposal can achieve better image recovery compared to existing works, and still be compatible to standard decoder. Overall, the proposal provides the best trade-off among the error resiliency, coding efficiency, and standard compatibility.