The MultiIndic22MT 2024 Shared Task.

Development set: We provide FLORES-22 Indic dev set an extended version of FLORES-200 dev set spanning all 22 aforementioned languages.

Public Test set: We will evaluate on the IN22-Gen and IN22-Conv test sets. These are English original test sets. As we have always done, these test sets are public but we trust participants to not fine-tune on them for fairness and correctness.

Hidden Test set: Additionally, we will evaluate on a hidden test set which is Indic original. This spans 13 of the 22 languages and will be released a week before the end of the shared task deadline.

Update - 12th July, 2024: The hidden test set is now available at Source Original Test Set. The references have not been provided unlike the public test sets. Please translate from Indic to English and then submit your translations as per the submission instructions below.

We recommend using the BPCC dataset (Gala et al., 2023) which spans all 22 languages. Additional details of the dataset are present in the github repo. Note that one may pivot via English to obtain Indic-Indic parallel corpora.

We additionally recommend the use of monolingual data from Varta, IndicCorp v2 and Sangraha corpora.

Please check with the organizers once before using any other data sources.

IndicTrans2 one-to-many - large hf, distilled hf, large fairseq, distilled fairseq

IndicTrans2 many-to-one - large, distilled, large fairseq, distilled fairseq

IndicTrans2 many-to-many - large, distilled, large fairseq, distilled fairseq

mT5 - You may use models of any size. hf, github

IndicBart - hf unified script, hf separate script, github

VartaT5 - hf

BLOOM - hf

Gemma - hf

If you want to use any other LLM, please feel free but let us know in advance.

HuggingFace scripts/notebooks might be the most convenient

Fairseq v1, v2, lora and distillation fork

YANMTT

open-instruct and its fork for Indic languages if you plan to fine-tune LLMs.

Constrained submission: If you use the data and models mentioned above the your submission will be considered as constrained.

Unconstrained submission: Any other data or models are used without confirmation from the organizers will result in unconstrained submissions.

Participants may submit up to 2 systems, one Primary (ranked) and one Contrastive (unranked, optional).

For example if your team name is Garuda and you are making an Unconstrained, Primary submission on the Hidden test set then the zip file will be: Garuda-Unconstrained-Primary-Hidden.zip.

Under the zip file there should be files for each direction in the format: (srccode)-(tgtcode).txt.

The codes should be as per the codes listed here. Thus if you are submitting Hindi to English and Punjabi to Sindhi (Devanagari script) translations then the file names will be: hin_Deva-eng_Latn.txt and pan_Guru-snd_Deva.txt.

We recommend following this script for evaluation, since we will do the same.