New Development Funcionality For The Asiya Suite Parameter Optimization With Mert
Project leader: Meritxell Gonzàlez, Cristina España-Bonet
Desirable skills for participants: C++, perl
Asiya is an open source toolkit for MT evaluation an meta-evaluation. The metric repository incorporates the latest versions of most popular metrics, operating at different linguistic dimensions (lexical, syntactic, and semantic) and based on different similarity assumptions (precision, recall, overlap, edit rate, etc.). The algorithms used during the development step in Moses optimize towards a small number of different metrics such as BLEU, TER or METEOR. Other metrics could be added to the process. However, they have to be explicitly integrated in the code. The purpose of this project is to decouple the metric used from the optimization algorithm.
The idea behind is to create an abstact metric class within Moses' MERT module that uses the set of scores calculated by Asiya. No hard integration of the two systems will be needed, but an API to communicate each other. In principle, we will include a new option within the Asiya suite to run MERT (or any other) with the specified metric-set. We will also add the required configuration to run Asiya from the MERT module.
We believe that this can be a quite simple project that can provide a great benefit to everybody.
System setup
We have set up a repository to work on this project:
At the moment there is a unique user called reader/reader to checkout. We will grant +w permission to the new developers.
The only requirements to run Asiya are a linux/macosx machine, and perl5.