Confidence metrics

Project leaders: Patrik Lambert, Holger Schwenk and Loïc Barrault

In many scenarios we need to evaluate the quality of a translated sentence, but no reference translation is available. For example, in the "wikitrans" scenario (, in which users can propose corrections to automatic translations, it is useful to determine whether the proposed correction is valid or not. Automatic translations can be added to a parallel corpus to adapt it to another domain (Schwenk 2008). In this scenario some confidence on the quality of the translation is also useful to discard incorrect translations. Several confidence metrics have been proposed. The aim of the project is to implement some of them.

Team: Raphael Payen, Haithem Afli, Andreas Kirkedal, Loïc Barrault

DAY1 : Survey

S. Raybaud, C. Lavecchia, D. Langlois, K. Smaïli Word- and Sentence-level Confidence Measures for Machine Translation

N. Ueffing and H. Ney, Word-Level Confidence Estimation for Machine Translation

J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, N. Ueffing Confidence Estimation for Machine Translation

C. B. Quirck Training a Sentence-Level Machine Translation Confidence Measure

L. Specia Estimating the Sentence-Level Quality of Machine Translation Systems

DAY2: Ideas

  • Use inter and intra mutual information
  • Use forward and backward LM
  • Use system score
  • Combine these scores in a linear model

Got some code from S. Raybaud. There is no documentation ... we need to understand that code!!!


  • Jesus Gimenez propose to evaluate CM consistency using WMT data and human ranking for those data. That sounds a good idea.
  • Sylvain Raybaud sends us an email to help us to use his programs. We have to decide whether we will use them or write some new ones.

DAY4: - tokenizing - reversing (thanks Andreas!) - nothing (thanks Loic)


 train_cmi -e corpus/news-train08.tok.en -f corpus/ --mi mifile -N

 train_imi -t corpus/news-train08.tok.en  --mi mifile -N

 cm-ngram -lm LMs/ -text corpus/news-train08.tok.en
Page last modified on September 17, 2010, at 07:06 PM