ANNOUNCEMENTS
-
2026-07-29: Submission deadline extended to 2 August
-
2026-07-25: Submissions are open for Task 1! Check the Codabench competition page!
-
2026-07-24: Added submission procedure (Codabench link coming soon!)
-
2026-07-23: Test set released!
-
2026-07-20: Test set and submission formats released
-
2026-07-16: Updated link to the MQM 2025 evaluation data
-
2026-06-06: Updated task description including detailed evaluation information
-
2026-04-29: Detailed description of the task released
TASK OVERVIEW
The goal of this task is to identify translation errors and assign a severity label to each error. Given a source text, its translation, and optionally a reference translation (available only for a subset of language directions), participants are asked to:
-
Detect error spans in the form of start and end indices,
-
Classify each span according to its severity:
-
Minor: Imperfections or stylistic issues that do not impact the core message (e.g., awkward phrasing).
-
Major: Confuses meaning, misrepresents the source, or violates the message (e.g., incorrect information, confusing wording).
-
To broaden participation and facilitate a more comprehensive analysis of current MT evaluation capabilities, this year, we are also introducing an automatic opt-in for participants across tasks. Submissions to this task will automatically be converted to participate in Task 2 (Segment-Level Quality Score Prediction) and Task 3 (Detection of Error-Free Segments). See the “Submissions” section below for details.
Language Coverage and Challenge Sets
This task uses the same data and a subset of the language pairs from the WMT 2026 General MT task (additional information below).
We strongly encourage all Task 1 participants to run and submit their system output on the evaluation data for our full set of 21 language pairs. However, we recognize that this may not be computationally feasible for all participants. If you are unable to run on the complete list of language pairs, please prioritize the eight translation directions listed in the general description of the shared task.
Participants are also expected to run their automatic systems on collected “challenge sets” that focus on particular linguistic phenomena, domains, or even non-WMT language pairs of interest to the developers of the sets. The predicted error spans will be returned to the developers of each set for further analysis. See the detailed page on the challenge set subtask for further information.
DATA
Test Data
The ground-truth error spans are derived from human annotations collected in the WMT General MT shared task. This year, human annotations will be collected using a contrastive ESA methodology via the Pearmut annotation tool. For additional information and to check the official list of translation directions, please refer to their website.
The test set will consist of a collection of documents, each divided into segments, where a segment can be a single- or multi-sentence unit of text. Participants may choose to perform error detection using only the provided segments or to incorporate the broader document context.
We will release one unified test set across Tasks 1, 2, and 3. It will use the following JSON-lines format:
{
"item_id": "setID_###_srcLang_###_tgtLang_###_domain_###_docID_###_segID",
"src": "sourceText",
"ref": {
"text": "referenceText",
"type": "referenceType"
},
"hyps": {
"systemName1": "translationText1",
"systemName2": "translationText2",
...,
"systemNameN": "translationTextN"
},
"resources": {
"screenshot": "screenshotFile",
"video": "videoFile",
"asr": "asrSourceText"
}
}The value of the ref.type field indicates whether the accompanying reference was produced by native human translation ("human"), human post-editing of MT output ("postedit"), or automatic generation ("pseudo"). The reference type is consistent within each language pair.
When present, the resources structure supplies any additional multimodal inputs that participants may choose to take into account when assessing translation quality. Screenshots and video files are given as relative path names within the General MT task’s package of multimodal inputs. For segments that originated as audio, the same raw ASR transcript (possibly containing errors) that was made available to MT participants is provided. Note, however, that the src and ref contents are based on a corrected transcript, and the human evaluation will be also.
You may refer to the detailed translation instructions provided by the General MT task for each segment. This metadata expresses desired properties of a correct translation in each domain, such as the level of formality, replication of style or voice, etc. The annotation instructions are also available. Submissions are encouraged to make use of the provided information to more accurately judge each translation’s quality.
Training and Development Data
You are welcome to build your system from any desired training data, foundation model, etc. We therefore do not release any specific training or development corpora. However, labeled data from previous editions of WMT shared tasks is available to help you train and tune your system if you would like.
These resources are summarized below:
-
ESA WMT25: github.com/wmt-conference/wmt25-general-mt
-
ESA WMT24: github.com/wmt-conference/wmt24-news-systems
-
MQM WMT20 to WMT24: github.com/google/wmt-mqm-human-evaluation
-
MQM WMT25: data.statmt.org/wmt25/metrics-qetask2/
SUBMISSIONS
Submissions will be collected via Codabench; check here later for the link. You may submit up to two system variants to this shared task per participating organization or research group.
Your submission must include a JSON-lines file of predicted segment-level error annotations, where each line has the following format:
{
"item_id": "setID_###_srcLang_###_tgtLang_###_domain_###_docID_###_segID",
"task1_pred": {
"systemName1": {
"errors" : [
{
"start": start,
"end": end,
"severity": "<minor/major>",
"category": "<optional error category>"
},
...
],
"omission": "<null / minor / major>"
},
"systemName2": {...},
...,
"systemNameN": {...}
}
}For each segment and translation, participants are expected to provide a list of predicted error spans. Each error span identifies a single contiguous sequence of characters in the translation that is considered erroneous. Each span is defined by its start and end character indices, where the start index is inclusive and the end index is exclusive, and by a severity, which is either "minor" or "major". Valid spans must have 0 ≤ start < end ≤ len(translation).
The category field can be used to indicate an error category. Please note that the category field is optional and will not impact the evaluation in any way, but it might be used for further analyses; if it is not present, it will be simply treated as null.
In this evaluation task, only the translation is annotated. Thus, to indicate the presence of omission errors (content from the source that is missing from the translation), participants should use the omission field. The value of this field should be null if no omission errors are detected, "minor" if a minor omission is detected, and "major" if a major omission is detected.
For any language pair you attempt, you must provide predictions for all translations across all segments.
Submission Procedure
Step 1: Register your participation by creating an account on Codabench and by joining the WMT 2026 segment-level error detection and span annotation competition.
Step 2: Download the test set file containing source–target segment pairs for scoring. (The format of this file is explained in the "Test Data" section above.)
Step 3: Use your automatic system to score each of the source–target segment pairs for each language pair in which you wish to participate. Produce only a single output file per submission. If you choose not to complete a given language pair or a given challenge set, remove its content entirely from your output file.
Step 4: For each submission, download and fill in the "model card" questionnaire with information about your system, which participating team it’s associated with, resources used, etc.
Step 5: Submit your output to the shared task by uploading it to our Codabench competition by the end of the test period (2 August). You must upload a single zip file (of any name) containing two files:
-
predictions.jsonl: the file containing your predicted error spans and severities for each translation, formatted as illustrated above. -
model_card.json: the model card filled with your submission information.
The two component files must be located at the root directory of the zip. To accommodate failed, erroneous, or mistaken uploads, you may submit up to five times per day but not exceeding 10 times over the course of the test period.
Step 6: Ensure that your zip file was properly processed by Codabench. Its status should show as "Finished" within a few minutes, and numerical scores should appear for it on the leaderboard for all official language pairs in which you expect to have participated. Note that leaderboard scores are for verification purposes only: they are computed against annotations coming from a simple baseline and they do not reflect the true outcome of the shared task! However, if the reported MPP F-score is exceptionally low (<0.1), that may indicate that you made some error in running your system or assembling your JSONL file. Furthermore, if there is a clear problem processing your submission, its status should show as "Failed". In case of both failed and successful submissions, a processing log should be available for more detailed inspection.
Step 7: By the end of the test period (2 August), ensure that your team has no more than two submissions present in Codabench. Delete from the leaderboard any others that you do not wish us to consider as final entries to the shared task. To add or delete a submission from the leaderboard, go to "My Submissions". Each submission has an "Actions" column with the relevant buttons.
Contact wmt-qe-metrics-organizers@googlegroups.com in the event of any questions or difficulties with the submission procedure.
Automatic Opt-In
Unless you explicitly opt out, submissions to this task will also be automatically evaluated for Tasks 2 and 3.
-
Task 2: We will automatically convert the predicted error spans and their severities for each segment into a numerical quality score using an MQM-style weighting scheme. Major and Minor errors will be assigned weights of 10 and 2.5 points, respectively. The score for each translation is computed by subtracting the total penalty from 100 (that is, the score of a perfect translation under ESA). The minimum possible score is 0; any additional penalties that would reduce the score below zero are ignored. Translations annotated with a single Major error spanning the entire segment will be assigned a score of 0.
-
Task 3: We will automatically derive binary labels based on your predicted error spans. Segments where your system predicts no errors will be labeled as 1 (error-free), while segments with one or more predicted spans will be labeled 0 (contains errors).
Participants in Task 1 may either submit separately to Tasks 2 and 3 or have their Task 1 submissions automatically evaluated for those tasks. Participants may opt out of this automatic evaluation by explicitly stating that their Task 1 submission should be evaluated only for Task 1.
EVALUATION
The primary evaluation metric will be MPP with micro-averaging (arxiv.org/abs/2603.19921), modified to incorporate error severity.
Given two sets of hypotheses and ground truth error spans, MPP with micro-averaging comprises the following steps:
-
Derive a one-to-one matching between hypothesis and ground-truth error spans.
-
Compute precision and recall for each pair of matched errors based on their proportion of overlapping characters.
-
Derive corpus-level precision and recall by averaging per-span precision and recall.
-
Compute the final F-score from corpus-level precision and recall.
We modify the MPP formulation to incorporate severity information. Our severity-weighted MPP replaces equal per-span averaging with severity-aware weighting. Each predicted span contributes to precision in proportion to its severity, each gold span contributes to recall in proportion to its severity, and matched pairs receive overlap credit that is reduced when the predicted and gold severities disagree. As a result, minor false positives and false negatives are penalized less than major ones, while severity mismatches receive partial rather than full credit.
Implementation
We plan to use the MPP implementation from the following GitHub repository: github.com/sted19/span-mt-metrics-eval.
Given lists of predicted and reference error spans across multiple segments, we plan to use the following evaluation configuration:
from span_mt_metrics_eval import compute
predictions = [...]
references = [...]
compute(
predictions,
references,
measure="MPP",
matching="one_to_one",
matching_algorithm="optimal",
averaging="micro",
severity_weights={"minor": 0.25, "major": 1.0},
)If optimal matching is computationally infeasible, we may use greedy matching instead, which is also supported by the implementation with matching_algorithm="greedy".
BASELINES
We will include in the official results a number of baseline systems. We currently intend to run the following ones:
-
xCOMET
-
Gemma 4
-
Qwen 3.6
-
Gemini 3.6 Flash
-
GPT-5.6 Terra