Challenge Set Subtask

EMNLP 2026

ELEVENTH CONFERENCE ON
MACHINE TRANSLATION (WMT26)

28-29 October, 2026
Budapest, Hungary HOME

TRANSLATION TASKS:	GENERAL MT •︎ INDIC MT •︎ ARABIC-ASIAN MT •︎ CHINESE-SOUTHEAST ASIAN MT •︎ TERMINOLOGY •︎ MODEL COMPRESSION •︎ CREOLE MT •︎ VIDEO SUBTITLE TRANSLATION
EVALUATION TASKS:	MT TEST SUITES •︎︎ AUTOMATED MT EVALUATION
OTHER TASKS:	OPEN DATA •︎ MULTILINGUAL INSTRUCTION •︎ LIMITED RESOURCES LLM

Task description

While other participants focus on are worried about building stronger and better metrics, participants of this subtask have to build challenge sets that identify where metrics fail! Continuing the work on the challenge sets of previous years, the goal of this subtask is for participants to create test sets with challenging evaluation examples that current metrics do not evaluate well. The challenge sets can relate to any (or all) of the three other subtasks, i.e., explore challenges on error detection, span annotation, quality score prediction and detection of error-free segments. sentence-level quality estimation, fine-grained error-detection, or error correction (see below).

This subtask is organized into 3 rounds:

Breaking Round: Challenge set participants (Breakers) create challenging examples for metrics. They must send the resulting "challenge sets" to the organizers by July 16th.
Scoring Round: The challenge sets created by Breakers will be included in the test sets of the three main subtasks and thus sent to all system participants (Builders) to score on.
Analysis Round: Breakers will receive their data with all the metric scores for analysis.

Challenge set types

This year, we are inviting submissions of challenge sets for all three other subtasks:

Task 1: Segment-Level Error Detection and Span Annotation
Task 2: Segment-Level Quality Score Prediction
Task 3: Detection of Error-Free Segments

Examples

Example challenge sets from previous years can be found below:

Submission details

Challenge set participants can submit challenges for in any or all of the other sub-tasks.
Challenge sets in all possible language pairs are welcome, however, system participants may choose to exclude language pairs, other than the official shared task ones, that are not supported by their systems
A limit of 1M tokens (per participant, including all language pairs) is introduced to facilitate the processing by translation systems that have high computational resource requirements. Participants whowhich are severely hindered by the limit are encouraged to communicate with the organizers.
The shared task organisers will not shuffle or change the order of segments provided. The provided segments will be incorporated into the main test data of the respective subtask and provided to participants for scoring.

Note that, depending on the subtask(s) that you intend to apply your challenge set to, we expect you to select the respective options in the registration form, and we will incorporate your data with all selected subtasks.

Submission format

The submission format will follow the JSONL file structure used by WMT-GenMT with minor adaptations to accommodate the needs for the Automated MT Evaluation tasks. Each line should look similar to the following sample:

{
  "doc_id": "cs-de_DE_#_WhateverChallenge_#_blesk.cz.112043_#_0",
  "src_lang": "cs",
  "tgt_lang": "de_DE",
  "src": "Ina T. hodnotí sexy pokusy hvězd: Myslivcová přestřelila!",
  "tgt": {
    "WhateverChallenge_hyp1": "Ina T. bewertet sexy Auftritte der Stars: Myslivcová hat übertrieben!",
    "WhateverChallenge_hyp2": "Ina T. bewertet sexy Versuche der Stars: Myslivcová hat übertrieben!"
  },
  "ref": {
    "refA": "Ina T. bewertet die sexy Versuche der Stars: Myslivcová hat es diesmal übertrieben!"
  }
}

In the above sample we have added linebreaks and identation, in order to make it visible, but the submitted files should have one JSONL per line:

{"doc_id": "cs-de_DE_#_Whatever-Challenge_#_blesk.cz.112043_#_0", "src_lang": "cs", "tgt_lang": "de_DE", "src": "Ina T. hodnotí sexy pokusy hvězd: Myslivcová přestřelila!",  "tgt": {"Whatever-Challenge_hyp1": "Ina T. bewertet sexy Auftritte der Stars: Myslivcová hat übertrieben!",   "Whatever-Challenge_hyp2": "Ina T. bewertet sexy Versuche der Stars: Myslivcová hat übertrieben!"}, "ref": {"refA": "Ina T. bewertet die sexy Versuche der Stars: Myslivcová hat es diesmal übertrieben!"}}

One JSONL file per language pair should be submitted. The filename should be of the format <srclang>.<tgtlang>.<WhateverChallenge>.jsonl
ref contains the references and tgt the hypotheses.
It is possible to have zero (no ref attribute), one or more references (refA, refB etc.) per source. If no reference is given, the file may be scored only by evaluation systems which can operate without references (quality estimation).
One or more hypotheses per source are possible. Please keep the suffix hyp1 even if you have only one hypothesis.
The number of hypotheses and references in every file should be the same for all entries in that file.
Linebreaks within the text can be encoded with "\n".
The doc_id should be unique.
If possible please choose an abbreviation of up to 7 letters for the challenge set name in the relevant entries above.

Challenge set registration

Please register your submission with contacts and a short description here.

Challenge set submission

You can upload your submission in an archive file (preferably .tar.gz, or .zip) until July 16th (AoE) here.

ELEVENTH CONFERENCE ON MACHINE TRANSLATION (WMT26)