Extend Pro And K Best Mira To Other Error Metrics
Project leader: Colin Cherry
Desirable skills for participants: C++, Perl
PRO and k-best MIRA are currently available in Moses for tuning high-dimensional feature vectors in a batch setting. However, they are both hard-coded to use (different) sentence-level approximations to BLEU. This project would augment both systems to use arbitrary sentence-level error metrics.
PRO and k-best MIRA share a lot of code, and hence it should be easy to change both at once. They currently read in feature vectors paired with vectors of BLEU sufficient statistics. This project would augment those statistics with an optional pre-calculated sentence-level score, such as TER or METEOR, which could then be used to guide discriminative tuning in place of BLEU.
This promises to be a fairly straightforward project, but it would also be quite useful. Other than the alterations to the tuners and to mert-moses.pl, we would want to integrate as many error metrics as possible, and/or enable a clear API for adding new metrics. We are likely to have time to attempt something more ambitious afterward; for example, we could develop a system to automatically tune the learning hyper-parameters to match a new metric.