Task description

This task aims to evaluate systems on the translation of documents from the biomedical domain. The test data will consist of biomedical abstracts and it will address the following language pairs:

• English-Chinese and Chinese-English (en/zh, zh/en) Unfortunately, no zh/en test sets this year.
• English-French and French-English (en/fr, fr/en)
• English-German and German-English (en/de, de/en)
• English-Italian and Italian-English (en/it, it/en)
• English-Portuguese and Portuguese-English (en/pt, pt/en)
• English-Russian and Russian-English (en/ru, ru/en)
• English-Spanish and Spanish-English (en/es, es/en)

Data

ATTENTION!! We ask the participants to not download the Medline database by themselves in order to retrieve training data. Submissions that are derived from a model that was trained on the whole PubMed will be not considered in the evaluation.

Participants can rely on training (and development) data from various sources, for instance:

• The Biomedical Translation repository includes links to parallel corpora of scientific publications (en/pt, en/es, en/fr, en/de, en/zh, en/it, en/ru), among others
• The UFAL Medical Corpus (formerly HimLCorpus) includes medical text from various sources for many language pairs (en/es, en/de, en/fr, en/ro) HimL test sets can be used as the development sets for some language pairs (en/es, en/de, en/fr, en/ro)
• The Khresmoi development data can be used for some language pairs (en/es, en/de, en/fr).
• The UNCorpus contains training data for some languages (en/es, en/fr, en/zh)
• The MeSpEn corpus contains many parallel documents of en/es
• The Scielo full text corpus for en, es and pt
• The Brazilian Thesis and Dissertations for en/pt

Participants are also free to use out-of-domain data.

Evaluation

• Scientific abstracts:
• Chinese/English (both directions) Unfortunately, no zh/en test sets this year.
• French/English (both directions)
• German/English (both directions)
• Italian/English (both directions)
• Portuguese/English (both directions)
• Russian/English (both directions)
• Spanish/English (both directions)

Scientific abstracts:

For the test set of Medline abstracts, the format will be plain text files. The format will be the following:

DOC_ID	SENT_ID	SENTENCE_TEXT

• DOC_ID: sequential one, e.g. doc1, doc8, not the original PMID in Medline
• SENT_ID: a sequential number from 1 to n
• SENTENCE_TEXT: the sentence text to be translated by the participants

doc1	1	sentence_1
doc2	2	sentence_2
doc2	3	sentence_3
doc2	4	sentence_4
doc2	5	sentence_5
...
doc2	n	sentence_n
doc4	1	sentence_1
doc4	2	sentence_2
...

doc1	1	translated_sentence_1
doc2	2	translated_sentence_2
doc2	3	translated_sentence_3
doc2	4	translated_sentence_4
doc2	5	translated_sentence_5
...
doc2	n	translated_sentence_n
doc4	1	translated_sentence_1
doc4	2	translated_sentence_2
...

Submission Requirements

Please notice that, following general WMT policy explicitly enforced in other tasks, we will release all participants' submissions after this year's edition of the task to promote further studies.

Please register your team using this form. You will receive a mail with the confirmation of your registration. The link for submission site will be informed in this mail. Please register your team as soon as possible.

The Medline test files are available in the WMT'23 biomedical task Google Drive folder.

The format for the submission files should include the original test file name preceded by the team identifier (as registered in the form above) and the run number, following this example for the abstracts:

• The submission file for run 1 of the "ABC" team for the Medline scientific abstracts from English into Spanish should be called "ABC_run1_medline_en2es_es.txt".
• The submission file for run 3 of the "ABC" team for the Medline scientific abstracts from Spanish into English should be called "ABC_run3_medline_es2en_en.txt".

Each team will be allowed to submit up to 3 runs per test set. Please notice that the submission form will include questions about details of your methods, please inform as much details as possible, this is important for us.

Results

Important dates

All deadlines are in AoE (Anywhere on Earth).

Organisers