Modified rate distortion optimization using inter-block dependence for H.264/AVC intra coding

Jongmin You, Changryoul Choi, Jechang Jeong

Research output: Contribution to journalArticle

7 Scopus citations


H.264/AVC allows several prediction schemes including variable modes for intra and inter predictions. To take full advantage of all allowed modes, H.264 employs rate distortion optimization (RDO) as its mode decision method. However, RDO incorporated into the H.264 encoder assumes block independence because calculating rate distortion (RD) cost for all possible mode sets requires a huge number of computations. In contrast to this assumption, coding efficiency of intra prediction in the H.264 encoder is affected by reference pixels in neighboring blocks. Thus, RDO in the H.264 encoder has potential to improve coding performance by considering the accuracy of intra prediction for the next blocks during the mode decision process of the current block. In this paper, we propose new criteria for I4x4 mode decision and by using this criteria, the proposed method considers more mode sets to improve RD performance compared to the original RDO. Although the proposed method requires more computations compared to the original RDO, experimental results indicate that the proposed method noticeably improves coding efficiency compared to the original RDO.

Original languageEnglish
Pages (from-to)1383-1388
Number of pages6
JournalIEEE Transactions on Consumer Electronics
Issue number3
Publication statusPublished - 2008 Oct 28



  • H.264
  • Intra coding
  • Mode decision
  • Rate distortion
  • Video coding

Cite this