Quality Estimation Shared Task

The list below provides the language pairs covered this year and the annotation available for each language pair:

This year no training and validation datasets will be released for Task 1: Sentence-level quality estimation and Task 2: Fine grained error detection. Participants should use the datasets from the previous year’s shared task available at wmt-qe-task.github.io/wmt-qe-2023/. For the new task on quality-informed automatic post-editing (QEAPE), we release the training and validation data containing post-edits for English-Hindi and English-Tamil language pairs. In addition, there is synthetic data available on the GitHub repository, to further support the new Task 3. You can find it in the train_dev folders for each LP: github.com/WMT-QE-Task/wmt-qe-2024-data/tree/main/train_dev/Task_3.

We follow the trend of the previous years and we organise a sentence-level quality estimation subtask where the goal is to predict the quality score for each source-target sentence pair. Depending on the language pair, the participants will be asked to predict either the direct assessment (DA) score or the multi-dimensional quality metrics (MQM) score. In the case of English-Hindi participants can predict both scores. Detailed information about the language pairs, the annotation specifics, and the available training and development resources are available at wmt-qe-task.github.io/.

Find more details about the task in the Sentence-level quality estimation task description.

This is a word-level subtask where the goal is to predict the translation error spans as opposed to binary OK/BAD tasks as it is normally case with word-level quality estimation tasks. For this subtask we will use the error spans obtained from the MQM annotations. Participants will be asked to predict both the error span (start and end indices) as well as the error severity (major or minor) for each segment.

Find more details about the task in the Fine grained error detection task description.

This is a new task which proposes to combine quality estimation and automatic post-editing in order to correct the output of machine translation. In light of this, we invite participants to submit systems capable of automatically generating QE predictions for machine-translated text and the corresponding output corrections. The objective is to explore how quality estimates (possibly at different levels of granularity) can inform post-editing. For instance, global sentence-level QE annotations may guide more or less radical post-editing strategies, while word-level annotations can be used for fine-grained, pinpointed corrections. We also encourage approaches that leverage quality explanations generated by large language models. Although the task is focused on quality informed APE, we also allow participants to submit APE output without QE predictions to understand the impact of their QE system (but not outputs that do not contain the correction). Submission w/o QE predictions will also be considered official. The task focuses on two language pairs: English-Hindi and English-Tamil.

For more details see the QEAPE task description.