Diagnostic Evaluation Of Mt With Delic4mt
Project leader: Antonio Toral
Desirable skills for participants: java, perl, php, experience/interest in MT evaluation (beyond just BLEU!)
DELiC4MT is a piece of open-source software that allows to perform diagnostic evaluation of Machine Translation systems over linguistic checkpoints, i.e. source-language lexical elements and grammatical constructions specified by the user. We would like to improve and extend the tool in a number of ways. These are some possible tasks:
- Adapt the tool to a new language pair. This involves writing wrappers for PoS taggers, word align test sets, etc. [perl, or your favourite scripting language]
- Currently the metric used to evaluate checkpoints is recall-based. We'd like to add a precision-based metric. [java]
- Extend the tool to enable more fine-grained evaluation feedback (e.g. extract most frequent errors) [java, perl]
- Develop a web interface for the tool. Sample screenshots: http://www.computing.dcu.ie/~atoral/delic4mt/webdemo/ [php, codeigniter framework]
- Any other good idea you may have to improve/extend DELiC4MT!