OVERVIEW
This shared task focuses on automatic methods for estimating the quality of neural machine translation output at run-time, without relying on reference translations. It will cover estimation at sentence and word levels, as well as critical error detection. This year, the critical error detection task will also include a subtask where participants will be encouraged to provide explanations for the detected errors, as well as, suggest corrections.
Languages covered
-
English to German (MQM)
-
English to Spanish (MQM)
-
English to Hindi (MQM, DA, Post-edits)
-
English to Gujarati (DA)
-
English to Telugu (DA)
-
English to Tamil (DA, Post-edits)
This year no training and validation datasets will be released. Instead, the participants will use the datasets from the previous year’s shared task available at wmt-qe-task.github.io/
Tasks organised
Task 1: Sentence-level Quality Estimation
Task 2: Fine grained error detection
Task 3: Context-aware Evaluation and Correction
DEADLINES (tentative)
Test set ready to download |
July 2024 |
Submission deadline for QE task |
July 2024 |
Paper submission deadline to WMT |
TBA (follows EMNLP) |
WMT Notification of acceptance |
TBA (follows EMNLP) |
WMT Camera-ready deadline |
TBA (follows EMNLP) |
Conference |
12-13 November, 2024 |
All deadlines are in AoE (Anywhere on Earth). Dates are specified with respect to EMNLP 2024.
ORGANIZERS
-
Pushpak Bhattacharyya, Indian Institute of Technology Bombay, India
-
Frédéric Blain, Tilburg University, Netherlands
-
Rajen Chatterjee, Apple Inc, US
-
Sourabh Deoghare, Indian Institute of Technology Bombay, India
-
Catarina Farinha, Unbabel, Portugal
-
Markus Freitag, Google, US
-
Nuno M Guerreiro, Instituto de Telecomunicações, Unbabel, Instituto Superior Técnico, Centrale Supélec – Université Paris-Saclay
-
Diptesh Kanojia, University of Surrey, UK
-
Andre Martins: Instituto de Telecomunicações, Instituto Superior Técnico, Universidade de Lisboa (Lisbon ELLIS Unit), Unbabel
-
Matteo Negri, FBK, Italy
-
Constantin Orǎsan, University of Surrey, UK
-
Ricardo Rei, Unbabel, Portugal
-
José GC de Souza, Unbabel
-
Marco Turchi, Zoom, US
-
Chrysoula Zerva: Instituto de Telecomunicações, Instituto Superior Técnico, Universidade de Lisboa (Lisbon ELLIS Unit)
STAY IN TOUCH
-
Please register to our Google group to stay in touch.
-
Follow us on Twitter: @wmt_qe_task