MT Evaluation Subtask 1: Segment-Level Quality Score Prediction

The goal of this task is to predict a quality score for each source–target segment pair in the evaluation set. Depending on the language pair, participants will be asked to predict the numerical score deriving from the segment’s ESA or MQM annotation as provided by a human. Submissions will be evaluated and ranked based on their predictions’ correlations with these human-annotated scores at both the segment and corpus levels.

We welcome submissions covering any or all of the language pairs used in the WMT 2025 General MT task, as per the following:

The test sets for all language pairs except for English–Italian and English–Maasai will be provided with a reference translation, which therefore can be used by submitted systems as an additional input.

Participants will also run their automatic score prediction systems on collected “challenge sets” that illustrate particular linguistic phenomena, domains, or even non-WMT language pairs of interest to the developers of the sets. The predicted scores will be returned to the developers of each set for further analysis. See the detailed page on the challenge set subtask for further details.

Note that, for this task, the necessary output is only the segment-level numerical scores. Submitted systems are free, however, to derive their scores in any manner that appears useful. In particular, systems may first predict precise error spans (as in Task 2) and/or post-edits (Task 3) and use them as the basis for computing the numerical score. Such systems that perform multiple subtasks jointly may be entered in each track.

You are welcome to build your system from any desired training data, foundation model, etc. We therefore do not release any specific training data. However, labeled data from previous editions of the Metrics and/or QE shared tasks is available to help you train and tune your system if you would like. Those resources are summarized and linked below. Note, however, that the set of language pairs covered in prior years is not an exact match for this year’s: some have training data, while some are zero-shot.

For help in tuning, model selection, etc., we recommend using the human-annotated General MT 2024 test set as a development set. We will release an updated version of this set soon.

The official test set will consist of a collection of documents, each divided into segments, to allow systems to evaluate quality in context. The segments, however, will consist of longer multi-sentence units of text. Contents of the challenge sets will vary. See below for details on the file format.

Evaluation sets are now available in Github. The single file containing both official and challenge sets is mteval-task1-test25.tsv.gz.

You may submit up to three system outputs to the shared task: one primary submission and up to two secondary submissions. Primary systems will feature in our official results, while secondary systems will be included in supplemental analysis.

Step 1: Register your participation by creating an account on Codabench and joining the WMT 2025 segment-level score prediction competition.

Step 2: At the start of the test week (24 July), download the evaluation set file containing source–target segment pairs for scoring.

Step 3: Use your automatic system to score each of the segments (and, optionally, each of the systems) for each language pair in which you wish to participate. Please ensure that your output files conform to the expected names and formats as defined above. If you choose to not participate in a given language pair or on a given challenge set, remove its content entirely from your output files.

Step 4: Submit your system to the shared task by uploading scored test sets to our Codabench competition by the end of the test week (31 July). Upload a single zip file (of any name) containing your output TSVs, which must be at the root directory of the zip. To accommodate failed, erroneous, or mistaken uploads, you may submit up to five times per day but not exceeding 10 times over the course of the test week.

Step 5: Ensure that your zip file was properly processed in Codabench. Its status should show as "Finished" within a minute, and numerical scores should appear for it on the leaderboard for all language pairs in which you expect to have participated. (Note that leaderboard scores are for verification purposes only and are computed against pseudo-gold-standard annotations. However, if the reported correlations are exceptionally weak, that may indicate that you made some error in running your system or assembling your TSV file.) If there is a clear problem processing your submission, its status should show as "Failed"; a processing log should be available for more detailed inspection.

Step 6: Also by the end of the test week (31 July), provide details about your system by filling in the participant form. In particular, be sure to give the Codabench IDs for your designated primary versus any secondary submissions. Completion of this form is essential to proper participation and an accurate analysis of the shared task results.

Contact wmt-qe-metrics-organizers@googlegroups.com in the event of any questions or difficulties with the submission procedure.

We will evaluate the quality of automatic score prediction on the official test set at both the segment level and the corpus (system) level. Evaluation will be carried out according to a combination of standard and more recently proposed meta-evaluation methodologies to assess correlation with human judgements.

At the segment level, we will use tie-calibrated pairwise accuracy (acc*_eq) as our primary meta-evaluation metric; we will additionally compute Pearson’s r and Kendall’s τ as secondary metrics. These correlations with human judgements will be computed per source segment and then averaged over the entire test set.

At the corpus level, we will use soft pairwise accuracy (SPA) as our primary meta-evaluation metric; we will again compute Pearson’s r and Kendall’s τ as secondary metrics. Because of the segment-level resampling required as part of the SPA computation, submissions that derive their corpus-level scores from a method other than an arithmetic averaging of the segment-level scores should be sure to indicate this fact on the participant form.

Scored challenge sets will be returned to the individual challenge set developers for evaluation and analysis. Performance on the challenge sets will not be tracked on the Codabench leaderboard or counted as part of the official results.

Note that the official results will be based on correlation with human judgements, which will not be complete until around September. At the time of test set submission (24–31 July), we will provide on the Codabench leaderboard an expression of each system’s correlation with automatically derived pseudo-gold-standard judgements, using simple metrics. This will allow participants to confirm that their files were uploaded in a valid format and that the performance of their system was broadly reasonable. Displayed scores, however, are for these basic verification purposes only and do not reflect official results.

System description papers must be submitted for initial review by 14 August and in camera-ready format by 25 September. Participants are therefore advised to begin preparation of their papers prior to the test week, using analysis based on a chosen development set rather than the official shared task results.

We will include in Codabench and in the official results a number of baseline systems. The exact selection will be announced later.