Shared Task: Automated Translation Quality Evaluation Systems

The list below provides the language pairs covered this year (which fully parallel the languages covered by the General MT shared task):

Human assessments of translation quality, collected by the General MT shared task using an ESA-based protocol, will act as the “gold standard” for our shared task evaluation. For details, see the “Human Evaluation” section of the General MT shared task description.

Training and validation datasets will be made available for each of our three subtasks. See the detailed task descriptions (once available) for the list of specific resources available for each of the tasks.

Submissions for Tasks 1, 2, and 3 (described below) will be automated and conducted via Codabench. We will provide an automated mechanism for easing the evaluation of submitted systems for all three tasks: direct submissions to Task 1 will by default also be evaluated on Tasks 2 and 3 via automations developed by the organizing committee. Similarly, direct submissions to Task 2 will by default also be evaluated on Task 3 via a similar automation. Participants have the option of opting out of this automated evaluation as well as submitting separate direct systems to the three tasks. Details of this automation process will be posted here at a later date.

Participants to any shared task will be expected to contribute a one-paragraph description of their system(s) for inclusion in our overall Findings paper, along with a four- to six-page system description paper for inclusion in the wider WMT conference proceedings. See the main WMT website for paper submission information.

Task 1 is a segment-level subtask where the goal is to detect translation errors and identify the precise span of each error within the target-side translation along with its severity. For this subtask we use the error spans obtained from the ESA human annotations generated for the General MT primary task as the target “gold standard.” Participants are asked to predict both the error spans (start and end indices) as well as the error severities (major or minor) within each segment. Submissions will be evaluated and ranked based on their ability to correctly identify the presence of errors, correctly mark the spans of any identified errors, and correctly identify the severity of each of these errors. Detailed information about the annotation specifics, the evaluation criteria, and the available training and development resources will be available shortly.

Task 2 is similar to the corresponding task from last year and is largely an updated version of similar tasks from previous years’ QE and Metrics tracks. The goal of the segment-level quality prediction subtask is to predict a quality score for each source–target segment pair in the evaluation set. Participants this year will be asked to predict the ESA score. Submissions will be evaluated and ranked based on their prediction correlations with human-annotated scores at both the segment and system levels. Detailed information about the annotation specifics, the evaluation criteria, and the available training and development resources will be available shortly.

Task 3 is a new subtask focused on a concrete application of translation quality evaluation systems: detection of error-free segments. While traditional quality estimation focuses on continuous scores or fine-grained error spans, this task simplifies the objective to a binary classification: Is this translation error-free? The goal is to identify segments that can be published or consumed without further human intervention. Participants must predict a binary label (1 for Error-Free, 0 for Contains Errors) for a given set of source–target pairs. Submissions will be evaluated and ranked based on their alignment with gold labels derived from human annotations, measured by the Matthews Correlation Coefficient. Detailed information about the input and output specifics and the evaluation methodology will be available shortly.

While the first three tasks are focused on the development of stronger and better automated quality evaluation systems, the goal of this subtask is for participants to create test sets with challenging evaluation examples that current automated metrics and evaluation systems fail to identify or score correctly. This subtask is organized into three rounds:

This year we are inviting submissions of challenge sets for all three official subtasks (see detailed descriptions above). In addition, challenge sets can target languages beyond the official language pairs listed for Tasks 1–3, but note that evaluation system developers may opt out from evaluating languages other than the official ones. If you are interested in submitting a challenge set this year, the organizers request that you indicate your intentions by completing the sign-up form here.