LLMs with Limited Resources for Slavic Languages: MT and QA

This year, the MT task will focus on two translation directions: German→Upper Sorbian and German→Lower Sorbian. Both language pairs were considered in the previous editions of WMT Shared Tasks on Unsupervised MT and Very Low Resource Supervised MT.

For Upper and Lower Sorbian-German Machine Translation, we provide the data from the WMT2022 Unsupervised MT and Very Low Resource Supervised MT Shared Task for both languages.

In addition, we provide the following new datasets thanks to the Witaj-Sprachzentrum:

More details on the datasets are available on: github.com/TUM-NLP/llms-limited-resources2025/

The Leipzig Corpora Collection (Goldhahn et al., 2012) contains monolingual corpora for both Sorbian languages:

As the two Sorbian languages are from the West Slavic language family, Czech (for Upper Sorbian) and Polish (Lower Sorbian) are two closely related, better-resourced languages. Both Czech and Polish have been languages in previous MT Shared Tasks; cs→de is one of the language pairs in this year’s general MT task.

The Witaj-Sprachzentrum provides language certificates for both Upper and Lower Sorbian, from the A1 to C1 levels, according to the CEFR (Common European Framework of Reference for Languages) scheme. We will use a mix of questions from all five levels (A1, A2, B1, B2, and C1, from beginner to advanced) for our task.

These language certificates assess a candidate’s language proficiency according to four pillars: listening comprehension, reading comprehension, written expression, and oral expression. For this Shared Task, we will use questions from the reading and listening parts using reference transcription of the audio material for the latter.

For each language level, there are different formats of exercises; this means that the question types in A1 are different from those in C1, for instance. If the beginner levels have true or false questions regarding a small text, the advanced exercises consist of multiple-choice questions with longer texts and statements.

For QA, the datasets are available at the following links:

Machine Translation: The MT task for Ukrainian will be focused on the English→Ukrainian and Czech→Ukrainian directions. The development data is the compilation of WMT datasets from 2022-2024 editions. For training, we recommend you refer to the WMT2025 main track setup with the details on various datasets useful for WMT systems fine-tuning.

Question Answering: The questions are taken from the Ukrainian External Independent Evaluation (called ЗНО/ZNO in Ukrainian) from various subjects: language, literature, history, geography, and other general knowledge. The training and development data is compiled from UNLP2024 Share Task: huggingface.co/datasets/osyvokon/zno

External datasets can be used on top of the provided corpora. For fairness and reproducibility, they should, however, be publicly available.

The output format for the Shared Task is a standardised JSONL file across languages and tasks, with four fields per instance:

Below is the description of the four fields:

Moreover, please note that the Upper and Lower Sorbian QA outputs should be concatenated into one file in order of difficulty (A1, A2, B1, B2, C1).

To make the output format conversion easier, we provide two resources. First, dummy outputs are present in the following folder of our GitHub repository: github.com/TUM-NLP/llms-limited-resources2025/tree/main/dummy_submission . Second, if you are using our fork of lm-evaluation-harness (see above), the conversion script has been updated to the output format: github.com/TUM-NLP/wmt25-lrsl-evaluation/blob/main/testphase-eval/convert_output_formats.py .

Summary of the modifications to make in the output:

For a full participation in our Shared Task, there will be seven files: 3 files for the Ukrainian track (CS-UK, EN-UK MT & UK QA), 2 files each for Upper and Lower Sorbian (DE-DSB|HSB MT & DSB|HSB QA).

Thanks to the main Shared Task organisers, the submissions for our Shared Task can also be handled by the OCELoT platform: ocelot-wmt.azurewebsites.net/

Instructions:

To check whether your submissions are correctly taken into account by the system, OCELoT displays some automatic metrics: BLEU and chrF for MT and accuracy for QA (except for CS-UK MT). Please note that these leaderboards are not the final ones; we will provide the final ranking shortly after the submissions phase is closed.

For better reproducibility, we highly recommend providing a link to your model in the system description paragraph by uploading it to HuggingFace (and mentioning external public datasets, when used).

Managed on SoftConf: softconf.com/emnlp2025/wmt2025/