MT Evaluation Subtask 2: Segment-Level Quality Score Prediction

You are welcome to build your system from any desired training data, foundation model, etc. We therefore do not release any specific training or development corpora. However, labeled data from previous editions of the Metrics, QE, and Evaluation shared tasks is available to help you train and tune your system if you would like. Those resources are summarized and linked below. Note, however, that the set of language pairs covered in prior years is not an exact match for this year’s: some have training data, while some are zero-shot.

The official test set will consist of a collection of documents, each divided into segments, to allow systems to evaluate quality in context. The segments, however, will consist of longer multi-sentence units of text. (By contrast, segments in 2024 data and earlier consisted of individual sentences.) Contents of the challenge sets will vary.

We release one unified test set across Tasks 1, 2, and 3. It can be found here. It uses the following JSON-lines format:

The value of the ref.type field indicates whether the accompanying reference was produced by native human translation ("human"), human post-editing of MT output ("postedit"), or automatic generation ("pseudo"). The reference type is consistent within each language pair.

When present, the resources structure supplies any additional multimodal inputs that participants may choose to take into account when assessing translation quality of the hypotheses. Screenshots and video files are given as relative path names within the General MT task’s package of multimodal inputs. For segments that originated as audio, the same raw ASR transcript (possibly containing errors) that was made available to MT participants is provided. Note, however, that the src and ref contents are based on a corrected transcript, and the human evaluation will be also.

You may refer to the detailed translation instructions provided by the General MT task for each segment. This metadata expresses desired properties of a correct translation in each domain, such as the level of formality, replication of style or voice, etc. The annotation instructions provided to the human annotators are also available. Submissions are encouraged to make use of the provided information to more accurately judge each translation’s quality.

Step 1: Register your participation by creating an account on Codabench and by joining the WMT 2026 segment-level score prediction competition.

Step 2: Download the test set file containing source–target segment pairs for scoring. (The format of this file is explained in the "Test Data" section above.)

Step 3: Use your automatic system to score each of the source–target segment pairs for each language pair in which you wish to participate. Produce only a single output file per submission. If you choose not to complete a given language pair or a given challenge set, remove its content entirely from your output file.

Step 4: For each submission, download and fill in the "model card" questionnaire with information about your system, which participating team it’s associated with, resources used, etc.

Step 5: Submit your output to the shared task by uploading it to our Codabench competition by the end of the test period (2 August). You must upload a single zip file (of any name) containing your JSONL output file (which must be called segments.jsonl) and model card (which must be called model_card.json). The two component files must be located at the root directory of the zip. To accommodate failed, erroneous, or mistaken uploads, you may submit up to five times per day but not exceeding 10 times over the course of the test period.

Step 6: Ensure that your zip file was properly processed by Codabench. Its status should show as "Finished" within a minute, and numerical scores should appear for it on the leaderboard for all official language pairs in which you expect to have participated. Note that leaderboard scores are for verification purposes only: they are computed against pseudo-gold-standard annotations that do not reflect the true outcome of the shared task! However, if the reported correlations are exceptionally weak, that may indicate that you made some error in running your system or assembling your JSONL file. If there is a clear problem processing your submission, its status should show as "Failed"; a processing log should be available for more detailed inspection.

Step 7: By the end of the test period (2 August), ensure that your team has no more than two submissions present in Codabench. Delete any others that you do not wish us to consider as final entries to the shared task.

Contact wmt-qe-metrics-organizers@googlegroups.com in the event of any questions or difficulties with the submission procedure.

To broaden participation and facilitate a more comprehensive analysis of current MT evaluation capabilities, we are introducing an automatic opt-in for participants across subtasks.

Unless you choose to opt out, your submission to this Task 2 will be evaluated as a submission to Task 3 (detection of error-free segments) as well. For this, you will be asked upon submission to provide a custom threshold for your model’s scores as well as an indication of whether the boundary is exclusive (>) or inclusive (≥). Any segment where your model score passes the threshold will be classified as error-free; all others will be classified as containing errors.

Submissions to Task 1 (automated error detection and span annotation) will likewise be automatically converted to participate in this Task 2. Task 1 participants may also submit up to two standalone systems to Task 2: this allows you to test specialized score-prediction techniques that may differ from your approach to error span detection. If you however wish for the same Task 1 system to be used for Task 2 via automatic conversion, there is no need to submit it twice.