Challenge Set Subtask

While other participants are worried about building stronger and better metrics, participants of this subtask have to build challenge sets that identify where metrics fail! Inspired by the challenge sets of previous years (Link_to_metrics, Link_to_QE), the goal of this subtask is for participants to create test sets with challenging evaluation examples that current metrics do not evaluate well. The challenge sets can relate to any (or all) of the three main subtasks, i.e., explore challenges on sentence-level quality estimation, fine-grained error-detection, or error correction (see below).

This subtask is organized into 3 rounds: 1. Breaking Round: Challenge set participants (Breakers) create challenging examples for metrics. They must send the resulting "challenge sets" to the organizers by July 5th. 2. Scoring Round: The challenge sets created by Breakers will be included in the test sets of the three main subtasks and thus sent to all participants (Builders) to score on. 3. Analysis Round: Breakers will receive their data with all the metric scores for analysis.

This year, we are inviting submissions of challenge sets for all three official subtasks: Task 1: Segment-level quality score prediction Task 2: Word-level error detection and span annotation Task 3: Quality-informed Segment-level Error Correction

Example challenge sets referring to task 1 from previous years can be found below.

Breakers are responsible for generating the data for their challenge sets: providing their own source segments, one or more target hypotheses, optionally one or more sets of references, and segment-level metadata.

We are adding below an updated challenge set format.

Please name the files:

respectively.

'langpair' should be similar to 'en-cs' (using the official ISO 639-1 two-letter codes). It is also possible to submit a (compressed) archive. You can find here an example comprising part of the WMT QE 2024 challenge sets.

The challenge set developers are encouraged to organize the challenge set(s) in a way that better suits their analysis, as the shared task organisers will not shuffle or change the order of segments provided. The provided segments will be incorporated to the main test data of the respective subtask and provided to participants for scoring.

Note that depending on the subtask(s) that you intend to apply your challenge set to, we expect you to select the respective options in the registration form and we will incorporate your data with all selected subtasks.