



\section{
\label{advanced-domain.tex}
Domain Adaptation}


\subsection{
Contents}






      - Translation Model Combination (Section \ref{advanced-domain.texntoc1}) 

      - Linear Interpolation and Instance Weighting (Section \ref{advanced-domain.texntoc2}) 

      -  Fill-up Combination (Section \ref{advanced-domain.texntoc3}) 

      -  Back-Off Combination (Section \ref{advanced-domain.texntoc4}) 

      - OSM Model Combination (Interpolated OSM) (Section \ref{advanced-domain.texntoc5}) 

      - Online Translation Model Combination (Multimodel phrase table type) (Section \ref{advanced-domain.texntoc6}) 

      - Online Computation of Translation Model Features Based on Sufficient Statistics (Section \ref{advanced-domain.texntoc7}) 

      - Alternate Weight Settings (Section \ref{advanced-domain.texntoc8}) 

      - Modified Moore-Lewis Filtering (Section \ref{advanced-domain.texntoc9})\\





\subsection{
\label{advanced-domain.texntoc1}Translation Model Combination}


You can combine several phrase tables by linear interpolation or instance weighting using the script 
{\bf {\em contrib/tmcombine/tmcombine.py}}, 
or by fill-up or back-off using the script 
{\bf {\em contrib/combine-ptables/combine-ptables.pl}}.



\subsubsection*{
\label{advanced-domain.texntoc2}Linear Interpolation and Instance Weighting}


Linear interpolation works with any models; for instance weighting, models need to be trained with the option {\tt -write-lexical-counts} so that all sufficient statistics are available.
You can set corpus weights by hand, and instance weighting with uniform weights corresponds to a concatenation of your training corpora (except for differences in word alignment).



You can also set weights automatically so that perplexity on a tuning set is minimized.
To obtain a tuning set from a parallel tuning corpus, use the Moses training pipeline to automatically extract a list of phrase pairs.
The file {\tt model/extract.sorted.gz} is in the right format.



An example call: (this weights {\tt test/model1} and {\tt test/model2} with instance weighting ({\tt -m counts}) and {\tt test/extract} as development set for perplexity minimization, and writes the combined phrase table to {\tt test/phrase-table\_test5})






\begin{SaveVerbatim}{myverb}
python tmcombine.py combine_given_tuning_set test/model1 test/model2 \\    -m counts -o test/phrase-table_test5 -r test/extract
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

More information is available in (Sennrich, 2012 EACL)\footnote{\sf http://www.aclweb.org/anthology/E/E12/E12-1055.pdf} and {\tt contrib/tmcombine/README.md}.



\subsubsection*{
\label{advanced-domain.texntoc3}\label{advanced-domain.texFillUpCombination} Fill-up Combination}


This combination technique is useful when the relevance of the models is known a priori:
typically, when one is trained on in-domain data and the others on out-of-domain data.



Fill-up preserves all the entries and scores coming from the first model, and adds entries 
from the other models only if new.
Moreover, a binary feature is added for each additional table to denote the provenance
of an entry. 
These binary features work as scaling factors that can be tuned directly by MERT along with 
other models' weights.



Fill-up can be applied to both translation and reordering tables.



Example call, where ptable0 is the in-domain model:

\begin{SaveVerbatim}{myverb}
perl combine-ptables.pl --mode=fillup   \\         ptable0 ptable1 ... ptableN > ptable-fillup
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

More information is available in (Bisazza et al., 2011 IWSLT)\footnote{\sf http://www.mt-archive.info/IWSLT-2011-Bisazza.pdf} and {\em contrib/combine-ptables/README.md}.



\subsubsection*{
\label{advanced-domain.texntoc4}\label{advanced-domain.texBackOffCombination} Back-Off Combination}


An additional combination technique, called back-off, is available, which is a simplified version of fill-up (Section \ref{advanced-domain.texFillUpCombination}). The only difference is that back-off technique does not generate the binary feature denoting the provenance an entry. This is also the main advantage of back-off: the combined table (ptable-backoff) contains the exact number of scores of their combining tables (ptable0, ptable1, ... ptableN).



Example call, where ptable0 is the in-domain model:

\begin{SaveVerbatim}{myverb}
perl combine-ptables.pl --mode=backoff    \\         ptable0 ptable1 ... ptableN > ptable-backoff
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}










\subsection{
\label{advanced-domain.texntoc5}OSM Model Combination (Interpolated OSM)}


OSM model trained from the plain concatenation of in-domain data with large and diverse multi-domain data is sub-optimal. When
other domains are sufficiently larger and/or different than the in-domain, the probability distribution can skew away from the target domain resulting in poor performance. The LM-like nature of the model provides motivation to apply methods such as perplexity optimization for model weighting. The idea is to train OSM model on each domain separately and interpolate them by minimizing optimizing perplexity on held-out tuning set. To know more read Durrani et al. (2015)\footnote{\sf http://alt.qcri.org/\textasciitilde{}ndurrani/pubs/joint-models-domain.pdf}. 



{\bf Usage}



Provide tuning files as additional parameter in the settings. For example:






\begin{SaveVerbatim}{myverb}
interpolated-operation-sequence-model = "yes"
operation-sequence-model-order = 5
operation-sequence-model-settings = "--factor 0-0 --tune /path-to-tune-folder/tune_file --srilm-dir /path-to-srilm/bin/i686-m64"
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

This method requires word-alignment for the source and reference tuning files to generate operation sequences. This can be done using force-decoding of tuning set or by aligning tuning sets along with the training. The folder should contain files as $<$tune.de , tune.en , tune.align$>$. 



Interpolation script does not work with LMPLZ and will require SRILM installation.



\subsection{
\label{advanced-domain.texntoc6}Online Translation Model Combination (Multimodel phrase table type)}


Additionally to the log-linear combination of translation models, Moses supports additional methods to combine multiple translation models into a single virtual model, which is then passed to the decoder.
The combination is performed at decoding time.



In the config, add a feature PhraseDictionaryMultiModel, which refers to its components as follows:






\begin{SaveVerbatim}{myverb}
[mapping]
0 T 2 [or whatever the zero-based index of PhraseDictionaryMultiModel is]

[feature]
PhraseDictionaryMemory tuneable=false num-features=4 input-factor=0 output-factor=0 path=/path/to/model1/phrase-table.gz table-limit=20
PhraseDictionaryMemory tuneable=false num-features=4 input-factor=0 output-factor=0 path=/path/to/model2/phrase-table.gz table-limit=20
PhraseDictionaryMultiModel num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=interpolate lambda=0.2,0.8 components=PhraseDictionaryMemory0,PhraseDictionaryMemory1

[weight]

PhraseDictionaryMemory0= 0 0 1 0
PhraseDictionaryMemory1= 0 0 1 0
PhraseDictionaryMultiModel0= 0.2 0.2 0.2 0.2
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

As component models, {\tt PhraseDictionaryMemory}, {\tt PhraseDictionaryBinary} and {\tt PhraseDictionaryCompact} are supported (you may mix them freely). Set the key {\tt tuneable=false} for all component models; their weights are only used for table-limit pruning, so we recommend {\tt 0 0 1 0 0} (which means p(e$|$f) is used for pruning).



There are two additional valid options for PhraseDictionaryMultiModel, {\tt mode} and {\tt lambda}.
The only {\tt mode} supported so far is {\tt interpolate}, which linearly interpolates all component models, and passes the results to the decoder as if they were coming from a single model.
Results are identical to offline interpolation with {\tt tmcombine.py} and {\tt -mode interpolate}, except for pruning and rounding differences.
The weights for each component model can be configured through the key {\tt lambda}. The number of weights must be one per model,
or one per model per feature.



Weights can also be set for each sentence during decoding through {\tt mosesserver} by passing the parameter {\tt lambda}. See {\tt contrib/server/client\_multimodel.py} for an example.
Sentence-level weights override those defined in the config.



With a running Moses server instance, the weights can also be optimized on a tuning set of phrase pairs, using perplexity minimization.
This is done with the XMLRPC method {\tt optimize} and the parameter {\tt phrase\_pairs}, which is an array of phrase pairs, each phrase pair being an array of two strings.
For an example, consult {\tt contrib/server/client\_multimodel.py}.
Online optimization depends on the dlib library, and requires Moses to be compiled with the flag {\tt -{}-with-dlib=/path/to/dlib}.
Note that optimization returns a weight vector, but does not affect the running system.
To use the optimized weights, either update the {\tt moses.ini} and restart the server, or pass the optimized weights as a parameter for each sentence.



\subsubsection*{
\label{advanced-domain.texntoc7}Online Computation of Translation Model Features Based on Sufficient Statistics}


With default phrase tables, only linear interpolation can be performed online.
Moses also supports computing translation probabilities and lexical weights online, based on a (weighted) combination of the sufficient statistics from multiple corpora, i.e. phrase and word (pair) frequencies.



As preparation, the training option {\tt -{}-write-lexical-counts} must be used when training the translation model.
Then, use the script {\tt scripts/training/create\_count\_tables.py} to convert the phrase tables into phrase tables that store phrase (pair) frequencies as their feature values.






\begin{SaveVerbatim}{myverb}
scripts/training/create_count_tables.py /path/to/model/phrase-table.gz /path/to/model
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The format for the translation tables in the {\tt moses.ini} is similar to that of the Multimodel type, but using the feature type {\tt PhraseDictionaryMultiModelCounts}
and additional parameters to specify the component models. Four parameters are required: {\tt components}, {\tt target-table}, {\tt lex-f2e} and {\tt lex-e2f}. The files required for the first two are created by {\tt create\_count\_tables.py}, the last two during training of the model with {\tt -{}-write-lexical-counts}. Binarized/compacted tables are also supported (like for {\tt PhraseDictionaryMultiModel}). Note that for the target count tables, phrase table filtering needs to be disabled ({\tt filterable=false}).






\begin{SaveVerbatim}{myverb}
[mapping]
0 T 4 [or whatever the zero-based index of PhraseDictionaryMultiModelCounts is]

[feature]
PhraseDictionaryMemory tuneable=false num-features=3 input-factor=0 output-factor=0 path=/path/to/model1/count-table.gz table-limit=20
PhraseDictionaryMemory tuneable=false num-features=3 input-factor=0 output-factor=0 path=/path/to/model2/count-table.gz table-limit=20

PhraseDictionaryMemory tuneable=false filterable=false num-features=1 input-factor=0 output-factor=0 path=/path/to/model1/count-table-target.gz
PhraseDictionaryMemory tuneable=false filterable=false num-features=1 input-factor=0 output-factor=0 path=/path/to/model2/count-table-target.gz

PhraseDictionaryMultiModelCounts num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=instance_weighting lambda=1.0,10.0 components=PhraseDictionaryMemory0,PhraseDictionaryMemory1 target-table=PhraseDictionaryMemory2,PhraseDictionaryMemory3 lex-e2f=/path/to/model1/lex.counts.e2f,/path/to/model2/lex.counts.e2f lex-f2e=/path/to/model1/lex.counts.f2e,/path/to/model2/lex.counts.f2e

[weight]
PhraseDictionaryMemory0= 1 0 0
PhraseDictionaryMemory1= 1 0 0
PhraseDictionaryMemory2= 1 
PhraseDictionaryMemory3= 1
PhraseDictionaryMultiModelCounts0= 0.00402447059454402 0.0685647475075862 0.294089113124688 0.0328320356515851
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

Setting and optimizing weights is done as for the Multimodel phrase table type, but the supported modes are different. The weights of the component models are only used for table-limit pruning, and the weight {\tt 1 0 0}, which is pruning by phrase pair frequency, is recommended.



The following modes are implemented:

\begin{compactitem}
\item {\tt instance\_weighting}: weights are applied to the sufficient statistics (i.e. the phrase (pair) frequencies), not to model probabilities. Results are identical to offline optimization with {\tt tmcombine.py} and {\tt -mode counts}, except for pruning and rounding differences.

\item {\tt interpolate}: both phrase and word translation probabilities (the latter being used to compute lexical weights) are linearly interpolated. This corresponds to {\tt tmcombine.py} with {\tt -mode interpolate} and {\tt -recompute-lexweights}.

\end{compactitem}

\subsection{
\label{advanced-domain.texntoc8}\label{advanced-domain.texalternateweightsettings}Alternate Weight Settings}


{\bf Note: this functionality currently does not work with multi-threaded decoding.}



You may want to translate different some sentences with different weight settings than others, due to significant differences in genre, text type, style, or even to have separate settings for headlines and questions.



Moses allows you to specify alternate weight settings in the configuration file, e.g.:






\begin{SaveVerbatim}{myverb}
[alternate-weight-setting]
id=strong-lm
Distortion0= 0.1
LexicalReordering0= 0.1 0.1 0.1 0.1 0.1 0.1
LM0= 1
WordPenalty0= 0
TranslationModel0= 0.1 0.1 0.1 0.1 0
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This example specifies a weight setting with the identifying name {\tt strong-lm}.



When translating a sentence, the default weight setting is used, unless the use of an alternate weight setting is specified with an XML tag:






\begin{SaveVerbatim}{myverb}
<seg weight-setting="strong-lm">This is a small house .</seg>
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This functionality also allows for the selective use of feature functions and decoding graphs (unless decomposed factored models are used, a decoding graph corresponds to a translation table).



Feature functions can be turned off by adding the parameter {\tt ignore-ff} to the identifier line (names of feature functions, separated by comma), decoding graphs can be ignored with the parameter {\tt ignore-decoding-path} (number of decoding paths, separated by comma).



Note that with these additional options all the capability of the previously (pre-2013) implemented "Translation Systems" is provided. You can even have one configuration file and one Moses process to translate two different language pairs that share nothing but basic features.



See the example below for a complete configuration file with exactly this setup. In this case, the default weight setting is not useful since it mixes translation models and language models from both language pairs.






\begin{SaveVerbatim}{myverb}
[input-factors]
0

# mapping steps
[mapping]
0 T 0
1 T 1

[distortion-limit]
6

# feature functions
[feature]
Distortion
UnknownWordPenalty
WordPenalty
PhraseDictionaryBinary name=TranslationModel0 num-features=5 \ 
path=/path/to/french-english/phrase-table output-factor=0 
LexicalReordering num-features=6 name=LexicalReordering0 \ 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 \ 
path=/path/to/french-english/reordering-table 
KENLM name=LM0 order=5 factor=0 path=/path/to/french-english/language-model lazyken=0
PhraseDictionaryBinary name=TranslationModel1 num-features=5 \ 
path=/path/to/german-english/phrase-table output-factor=0 
LexicalReordering num-features=6 name=LexicalReordering1 \ 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 \ 
path=/path/to/german-english/reordering-table 
KENLM name=LM1 order=5 factor=0 path=/path/to/german-english/language-model lazyken=0

# core weights - not used 
[weight]
Distortion0= 0
WordPenalty0= 0
TranslationModel0= 0 0 0 0 0
LexicalReordering0= 0 0 0 0 0 0
LM0= 0
TranslationModel1= 0 0 0 0 0
LexicalReordering1= 0 0 0 0 0 0
LM1= 0

[alternate-weight-setting]
id=fr ignore-ff=LM1,LexicalReordering1 ignore-decoding-path=1
Distortion0= 0.155
LexicalReordering0= 0.074 -0.008 0.002 0.050 0.033 0.042
LM0= 0.152
WordPenalty0= -0.097
TranslationModel0= 0.098 0.065 -0.003 0.060 0.156
id=de ignore-ff=LM0,LexicalReordering0 ignore-decoding-path=0
LexicalReordering1= 0.013 -0.012 0.053 0.116 0.006 0.080
Distortion0= 0.171
LM0= 0.136
WordPenalty0= 0.060
TranslationModel1= 0.112 0.160 -0.001 0.067 0.006
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

With this model, you can translate:






\begin{SaveVerbatim}{myverb}
<seg weight-setting=de>Hier ist ein kleines Haus .</seg>
<seg weight-setting=fr>C' est une petite maison . </seg>
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{advanced-domain.texntoc9}Modified Moore-Lewis Filtering}


When you have a lot of out-of-domain data and you do not want to use all of it, then you can filter down that data to the parts that are more similar to the in-domain data. Moses implements a method called modified Moore-Lewis filtering. The method basically train in-domain and out-of-domain language models, and removes sentence pairs that receive relatively low scores by the in-domain models. For more details, please refer to the following paper:



{\em Axelrod, Amittai and He, Xiaodong and Gao, Jianfeng}: {\bf Domain Adaptation via Pseudo In-Domain Data Selection}, Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing mentioned in Domain Adaptation, pdf\footnote{\sf http://aclweb.org/anthology//D/D11/D11-1033.pdf}, bib\footnote{\sf http://aclweb.org/anthology//D/D11/D11-1033.bib}.



The Moses implementation is integrated into EMS. You have to specify in-domain and out-of-domain in separate CORPUS sections (you can have more than one of each), and then set in the configuration file which out-of-domain corpora need to be filtered






\begin{SaveVerbatim}{myverb}
### filtering some corpora with modified Moore-Lewis
# 
mml-filter-corpora = giga
mml-before-wa = "-proportion 0.2"
#mml-after-wa = "-proportion 0.2"
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

There are two different places when to do the filtering, either before or after word alignment. There may be some benefits of having out-of-domain data to improve sentence alignment, but that may also be computationally to expensive. In the configuration file, you specify the proportion of the out-of-domain data that will be retained - in the example above 20% will be kept, 80% will be thrown out.












\begin{center}\textit{\small Subsection last  modified on October 09, 2015, at 11:33 AM}\end{center}
