Open Language Data Initiative

Workflow: Contributing a new language to FLORES+ and Seed typically involves starting from the original English data and translating it. Starting from a different language is also possible, but this choice should be clearly documented. Translations should be performed, wherever possible, by qualified, native speakers of the target language. We strongly encourage verification of the data by at least one additional native speaker.

Dataset card: dataset cards should be attached to new data submissions, detailing precise language information and the translation workflow that was employed. In particular, we ask participants to identify the language with both an ISO 639-3 individual language tag and a Glottocode. The script should be identified with an ISO 15924 script code.

Use of MT:

Data validation: Participants are strongly encouraged to provide experimental validation of the quality of the data they are submitting. For Seed data contributions, where applicable, this may include training a simple MT model and evaluating it on FLORES+.

License: As both FLORES+ and Seed are open datasets released under CC BY-SA 4.0, new contributions must also be released under this same license. By contributing data to this shared task, participants agree to have this data released under these terms.

Workflow: Data must not be synthetic, such as MT or LLM output. Other than that, there are no restrictions on the provenance of the data provided it is clearly identified along with its license. In order to ensure the quality of the data, contributors must verify that it is in the claimed language and free of issues such as encoding problems. If at all possible, this should be done by having one or more native speakers manually check a sufficiently large representative sample of the whole dataset.

Data card: data card should be attached to new data submissions, detailing precise language information and the translation workflow that was employed. In particular, we ask participants to identify the language with both an ISO 639-3 individual language tag and a Glottocode. The script should be identified with an ISO 15924 script code.

License: We encourage contributions under open licenses. At a minimum, data should be made available for research use.

A dataset accompanied by its corresponding data card, following the guidelines above and the in-depth instructions on the OLDI website. These should be shared with the organisers via email at <info@oldi.org>.

A system paper, prepared according to the WMT instructions and submitted via START.

Language overview. Provide background information on the language, highlighting dialectal and spelling variations, as well as the specific variety used in the submission. If applicable, mention any particular attributes which need special consideratino when developing NLP applications e.g. morphology, commonly confused languages, writing system(s). For under-resourced languages, include a review of existing datasets, reference materials, and other relevant resources.

Data collection. Offer an in-depth description of the data acquisition process. For example, when submitting a translated dataset, provide details such as the source language, number of translators, their expertise (native speakers, proficiency, professional experience), and whether any portion was independently reviewed by third parties.

Experimental validation. Provide experimental validation of the submitted data’s quality. For a seed translation dataset, this might involve training a translation model on new data and evaluating it on existing benchmark data, comparing it to pre-existing models or those trained with pre-existing data (where applicable). For monolingual data, this could include training and cross-validating language models or language identification (LID) classifiers on the new data.

Data sample. Provide a short excerpt of the data available in the dataset to demonstrate its format and content. This should take up no more than half a page. If possible, provide a translation in English.