Non-Repetitive Translation Task

This task focuses on lexical choice in machine translation, especially choice regarding repeated words in a source sentence. Generally, the repetition of the same words can create a monotonous or awkward impression in English, and it should be appropriately avoided. Typical workarounds in monolingual writing are to (1) remove redundant terms if possible (reduction) or (2) use alternative words such as synonyms as substitutes (substitution). These techniques are also observed in human translations. Here are some examples, which are all not contained in the test set of this task:

The goal of this task is to study how these techniques can be incorporated into machine translation systems to enrich lexical choice capabilities. From a practical standpoint, such capability would be important, for example, in news production, where high quality text that goes beyond robotic word-by-word translation is required.

Specifically, participants are required to control a machine translation system using reduction or substitution so that it does not output the same words for certain repeated words in a source sentence. The translation direction is Japanese to English.

The challenges underlying this task include the following:

We provide development and test sets for this task, which are referred to as the Jiji 2023 data and the Jiji 2024 data, respectively, in this description. In both data sets, all Japanese sentences contain some repeated words that are translated into English with reduction or substitution. We collected these data from Jiji Japanese–English news articles. Specifically, we first automatically created sentence pairs based on lexical similarities, and then manually selected instances suited for this task. These sentence pairs include not only one-to-one pairs but two-to-two pairs. Both the development and test sets contain raw and tagged parallel data. In the tagged data, repeated words in the source sentence and their counterparts in the target sentence are marked with tags, which indicates that these words are evaluation targets. Examples are as follows:

Note that not all words repeated in the source sentence are evaluation targets. This is because some words, such as proper nouns and technical terms, should be translated consistently, even if they are repeated in the sentence.

Tagged development data are provided to help tune the model during training. However, participants cannot use tagged test data and must use raw test data when submitting the system results. In this task, the systems must detect repeated words which can be reduced or substituted on their own.

To reduce the negative effects of imbalanced content in the source and target sentences, the Japanese sentences in the Jiji 2023 and 2024 data were manually translated from the English while preserving as much of the vocabulary of the original Japanese sentences as possible.

As for the training data, we also provide all the data from the WAT 2020 Newswire tasks, which were also constructed from Jiji news articles and have been continuously used in WAT from 2020. For simplicity, we refer to these data as the Jiji 2020 data. The main files in the Jiji 2020 data are as follows:

The above data are a regular parallel corpus and have not been annotated specifically for this task, but are in exactly the same domain as this task. Although "devc.txt" and "testc.txt" are not directly related to the evaluation of this task, these can be used to measure basic translation performance during training.

In addition, participants can also use any other publicly available corpora, such as the data from the general MT task in WMT, for training. When using external data, be sure to include an explanation about the data in your paper.

To obtain the train (Jiji 2020), development (Jiji 2023) and test (Jiji 2024) data, follow these steps:

Please anonymize any personal information when you include such text from the Jiji data in your papers and presentations.

System performance is evaluated using the total number of outputs that meet both acceptable translation adequacy and appropriate lexical choice. Both dimensions will be checked by human annotators, who will be assigned by the organizers.

Regarding the evaluation for lexical choice, the translations for the tagged source words are checked. (Again, the tags must be blinded to the system during inference.) Whether untagged repeated words are translated in a repetitive or non-repetitive way does not affect the lexical choice evaluation. Here, the technique (reduction or substitution) does not have to be consistent with that of the reference translation. In our preliminary investigations, we qualitatively studied the lexical choices of several translators, and observed cases where one translator chose substitution and another chose reduction. In addition, the systems do not have to choose the same words used in the reference, as long as the meaning is appropriate.

The determination of substitution or repetition is basically based on the word stem. For example, conversions between voice (e.g., "attack" and "be attacked"), tense (e.g., "study" and "studied") and parts of speech (e.g., "problematic" and "problem") are not considered to be substitutions. Conversions to idioms (e.g., "visit" and "pay a visit") are an exception and are handled as substitutions.

In addition, BLEU scores and style (i.e., non-repetitive or repetitive) estimations using a word aligner are also reported as a reference.

Please send your system translation as a file named "YOUR_TEAM_NAME.txt" by email to kinugawa.k-jg@nhk.or.jp and mino.h-gq@nhk.or.jp.

We are deeply grateful to Hidehiro Asaka and Takayuki Kawakami for providing the valuable data used in this research.
These research results were obtained from the commissioned research (No. 225) by National Institute of Information and Communications Technology (NICT), Japan.

Copyright©Jiji Press, Ltd. All rights reserved for the example sentences in this document.