Open Language Data Initiative

Workflow: Contributing a new language to FLORES+ and Seed typically involves starting from the original English data and translating it. Starting from a different language is also possible, but this choice should be clearly documented. Translations should be performed, wherever possible, by qualified, native speakers of the target language. We strongly encourage verification of the data by at least one additional native speaker.

Dataset card: dataset cards should be attached to new data submissions, detailing precise language information and the translation workflow that was employed. In particular, we ask participants to identify the language with both an ISO 639-3 individual language tag and a Glottocode. The script should be identified with an ISO 15924 script code.

Use of MT:

Data validation: Participants are strongly encouraged to provide experimental validation of the quality of the data they are submitting. For Seed data contributions, where applicable, this may include training a simple MT model and evaluating it on FLORES+.

License: As both FLORES+ and Seed are open datasets released under CC BY-SA 4.0, new contributions must also be released under this same license. By contributing data to this shared task, participants agree to have this data released under these terms.

Languages: For the translation quality annotation data track, we accept extensions or new datasets that include languages previously underrepresented in such data.

Openness: The source texts and human quality annotations should be opensource and, importantly, the translation systems involved should allow reusing their outputs for developing other systems.

Parallelism: The "multiway parallel data" requirement is not mandatory for this track, as the main phenomenon of interest, the variations in translation quality, is going to be language-specific anyway.

Annotation: The protocols for human annotation can be based on DA (direct assessment), MQM (multidimentional quality metrics), ESA (error span annotation), XSTS (cross-lingual semantic text similarity), or another methodology. This methodogoly should be clearly documented and, if the contribution is extending an exising dataset, the annotation protocol should be consistent with its original approach.

A dataset accompanied by its corresponding data card, following the guidelines above and the in-depth instructions on the OLDI website. These should be shared with the organisers via email at <info@oldi.org>.

A system paper, prepared and submitted according to the WMT instructions.

Language overview. Provide background information on the language, highlighting dialectal and spelling variations, as well as the specific variety used in the submission. If applicable, mention any particular attributes which need special consideration when developing NLP applications e.g. morphology, commonly confused languages, writing system(s). For under-resourced languages, include a review of existing datasets, reference materials, and other relevant resources.

Data collection. Offer an in-depth description of the data acquisition process. For example, when submitting a translated dataset, provide details such as the source language, number of translators, their expertise (native speakers, proficiency, professional experience), and whether any portion was independently reviewed by third parties.

Experimental validation. Provide experimental validation of the submitted data’s quality. For a seed translation dataset, this might involve training a translation model on new data and evaluating it on existing benchmark data, comparing it to pre-existing models or those trained with pre-existing data (where applicable). For benchmark data, this could include computing automatic metrics of data quality that are known to correlate with human judgements (e.g. applying machine translation quality estimation metrics to human translation outputs).

Data sample. Provide a short excerpt of the data available in the dataset to demonstrate its format and content. This should take up no more than half a page. If possible, provide a translation in English.