



\section{
\label{support-tools.tex}
Support Tools}


\subsection{
Overview}


Scripts are in the {\tt scripts} subdirectory in the source release in the Git repository.



The following basic tools are described elsewhere:

\begin{compactitem}
\item Moses decoder (Section \ref{tutorial.tex})

\item Training script {\tt train-model.perl} (Section \ref{factored-training.tex})

\item Corpus preparation {\tt clean-corpus-n.perl} (Section \ref{training-prepare.tex})

\item Minimum error rate training (tuning) {\tt mert-moses.pl} (Section \ref{tuning.tex})

\end{compactitem}

\subsection{
\label{support-tools.texntoc1}Converting Pharaoh configuration files to Moses configuration files}


Moses is a successor to the Pharaoh decoder, so you can use the same models that work for Pharaoh and use them with Moses. The following script makes the necessary changes to the configuration file:






\begin{SaveVerbatim}{myverb}
exodus.perl < pharaoh.ini > moses.ini
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{support-tools.texntoc2}Moses decoder in parallel}


Since decoding large amounts of text takes a long time, you may want to split up the text into blocks of a few hundred sentences (or less), and distribute the task across a Sun GridEngine cluster. This is supported by the script {\tt moses-parallel.pl}, which is run as follows:






\begin{SaveVerbatim}{myverb}
moses-parallel.pl -decoder decoder -config cfgfile -i input -jobs N [options]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Use absolute paths for your parameters (decoder, configuration file, models, etc.).






\begin{compactitem}
\item {\tt decoder} is the file location of the binary of Moses used for decoding

\item {\tt cfgfile} is the configuration fileofthe decoder

\item {\tt input} is the file to translate

\item {\tt N} is the number of processors you require

\item {\tt options} are used to overwrite parameters provided in {\tt cfgfile}\\Among them, overwrite the following two parameters for nbest generation (NOTE: they differ from standard Moses)
\begin{compactitem}
\item {\tt -n-best-file} output file for nbest list

\item {\tt -n-best-size} size of nbest list

\end{compactitem}

\end{compactitem}

\subsection{
\label{support-tools.texntoc3}Filtering phrase tables for Moses}


Phrase tables easily get too big, but for the translation of a specific set of text only a fraction of the table is needed. So, you may want to filter the translation table, and this is possible with the script:






\begin{SaveVerbatim}{myverb}
filter-model-given-input.pl filter-dir config input-file
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This creates a filtered translation table with new configuration file in the directory {\tt filter-dir} from the model specified with the configuration file {\tt config} (typically named {\tt moses.ini}), given the (tokenized) input from the file {\tt input-file}.



In the advanced feature section, you find the additional option of binarizing translation and reordering table, which allows these models to be kept on disk and queried by the decoder. If you want to both filter and binarize these tables, you can use the script:






\begin{SaveVerbatim}{myverb}
filter-model-given-input.pl filter-dir config input-file -Binarizer binarizer
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The additional {\tt binarizer} option points to the appropriate version of {\tt processPhraseTable}.



\subsection{
\label{support-tools.texntoc4}Reducing and Extending the Number of Factors}


Instead of the two following scripts, this one does both at the same time, and is better suited for our directory structure and factor naming conventions:

\begin{SaveVerbatim}{myverb}
reduce_combine.pl \ 
czeng05.cs \ 
0,2 pos lcstem4 \ 
> czeng05_restricted_to_0,2_and_with_pos_and_lcstem4_added
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{support-tools.texntoc5}Scoring translations with BLEU}


A simple BLEU scoring tool is the script {\tt multi-bleu.perl}:






\begin{SaveVerbatim}{myverb}
multi-bleu.perl reference < mt-output
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Reference file and system output have to be sentence-aligned (line X in the reference file corresponds to line X in the system output). If multiple reference translation exist, these have to be stored in seperate files and named {\tt reference0}, {\tt reference1}, {\tt reference2}, etc. All the texts need to be tokenized.



A popular script to score translations with BLEU is the NIST mteval script\footnote{\sf http://www.nist.gov/speech/tests/mt/2008/scoring.html}. It requires that text is wrapped into a SGML format. This format is used for instance by the NIST evaluation\footnote{\sf http://www.nist.gov/speech/tests/mt/2009/} and the WMT Shared Task evaluations\footnote{\sf http://www.statmt.org/wmt09/translation-task.html}. See the latter for more details on using this script.



\subsection{
\label{support-tools.texntoc6}Missing and Extra N-Grams}


Missing n-grams are those that all reference translations wanted but MT system did not produce. Extra n-grams are those that the MT system produced but none of the references approved.

\begin{SaveVerbatim}{myverb}
missing_and_extra_ngrams.pl hypothesis reference1 reference2 ...
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{support-tools.texntoc7}Making a Full Local Clone of Moses Model + ini File}


Assume you have a {\tt moses.ini} file already and want to run an experiment with it. Some months from now, you might still want to know what exactly did the model (incl. all the tables) look like, but people tend to move files around or just delete them.



To solve this problem, create a blank directory, go in there and run:

\begin{SaveVerbatim}{myverb}
clone_moses_model.pl ../path/to/moses.ini
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

{\tt close\_moses\_model.pl} will make a copy of the {\tt moses.ini} file and local symlinks (and if possible also hardlinks, in case someone deleted the original file) to all the tables and language models needed.



It will be now safe to run moses locally in the fresh directory.



\subsection{
\label{support-tools.texntoc8}Absolutizing Paths in moses.ini}


Run:

\begin{SaveVerbatim}{myverb}
absolutize_moses_model.pl  ../path/to/moses.ini > moses.abs.ini
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

to build an ini file where all paths to model parts are absolute. (Also checks the existence of the files.)



\subsection{
\label{support-tools.texntoc9}Printing Statistics about Model Components}


The script






\begin{SaveVerbatim}{myverb}
analyse_moses_model.pl moses.ini
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Prints basic statistics about all components mentioned in the moses.ini. This can be useful to set the order of mapping steps to avoid explosion of translation options or just to check that the model components are as big/detailed as we expect.



Sample output lists information about a model with 2 translation and 1 generation step. The three language models over three factors used and their n-gram counts (after discounting) are listed, too.






\begin{SaveVerbatim}{myverb}
Translation 0 -> 1 (/fullpathto/phrase-table.0-1.gz):
743193        phrases total
1.20  phrases per source phrase
Translation 1 -> 2 (/fullpathto/phrase-table.1-2.gz):
558046        phrases total
2.75  phrases per source phrase
Generation 1,2 -> 0 (/fullpathto/generation.1,2-0.gz):
1.04  outputs per source token
Language model over 0 (/fullpathto/lm.1.lm):
1     2       3   
49469 245583  27497
Language model over 1 (/fullpathto/lm.2.lm):
1     2       3   
25459 199852  32605
Language model over 2 (/fullpathto/lm.3.lm):
1     2       3       4       5       6       7  
709   20946   39885   45753   27964   12962   7524
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{support-tools.texntoc10}Recaser}


Often, we train machine translation systems on lowercased data. If we want to present the output to a user, we need to re-case (or re-capitalize) the output. Moses provides a simple tool to recase data, which essentially runs Moses without reordering, using a word-to-word translation model and a cased language model.



The recaser requires a model (i.e., the word mapping model and language model mentioned above), which is trained with the command:






\begin{SaveVerbatim}{myverb}
train-recaser.perl --dir MODEL --corpus CASED [--train-script TRAIN]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The script expects a cased (but tokenized) training corpus in the file {\tt CASED}, and creates a recasing model in the directory {\tt MODEL}. KenLM's lmplz is used to train language models by default; pass -{}-lm to change the toolkit.  



To recase output from the Moses decoder, you run the command






\begin{SaveVerbatim}{myverb}
recase.perl --in IN --model MODEL/moses.ini --moses MOSES [--lang LANGUAGE] [--headline SGML] > OUT
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

The input is in file {\tt IN}, the output in file {\tt OUT}. You also need to specify a recasing model {\tt MODEL}. Since headlines are capitalized different from regular text, you may want to provide an {\tt SGML} file that contains information about headline. This file uses the NIST format, and may be identical to source test sets provided by the NIST or other evluation campaigns. A language {\tt LANGUAGE} may also be specified, but only English ({\tt en}) is currently supported.



By default, EMS trains a truecaser (see below). To use a recaser, you have to make the following changes:

\begin{compactitem}
\item Comment out {\tt output-truecaser} and {\tt detruecaser} and add instead {\tt output-lowercaser} and {\tt EVALUATION:recaser}. 

\item Add {\tt IGNORE} to the {\tt [TRUECASING]} section, and remove it from the {\tt [RECASING]} section

\item Specify in the {\tt [RECASING]} section, which training corpus should be used for the recaser. This is typically the target side of the parallel corpus or a large language model corpus. You can directly link to a corpus already specified to the config file, e.g.,
tokenized = [LM:europarl:tokenized-corpus]

\end{compactitem}

\subsection{
\label{support-tools.texntoc11}Truecaser}


Instead of lowercasing all training and test data, we may also want to keep words in their natural case, and only change the words at the beginning of their sentence to their most frequent form. This is what we mean by truecasing. Again, this requires first the training of a truecasing model, which is a list of words and the frequency of their different forms.






\begin{SaveVerbatim}{myverb}
train-truecaser.perl --model MODEL --corpus CASED
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The model is trained from the cased (but tokenized) training corpus {\tt CASED} and stored in the file {\tt MODEL}.



Input to the decoder has to be truecased with the command






\begin{SaveVerbatim}{myverb}
truecase.perl --model MODEL < IN > OUT
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Output from the decoder has to be restored into regular case. This simply uppercases words at the beginning of sentences:






\begin{SaveVerbatim}{myverb}
detruecase.perl < in > out [--headline SGML]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

An SGML file with headline information may be provided, as done with the recaser.



\subsection{
\label{support-tools.texntoc12}Searchgraph to DOT}


This small tool converts Moses searchgraph ({\tt -output-search-graph FILE} option) to dot format. The dot format can be rendered using the graphviz\footnote{\sf http://www.graphviz.org/} tool dot.






\begin{SaveVerbatim}{myverb}
moses ... --output-search-graph temp.graph -s 3
# we suggest to use a very limited stack size, -s 3
sg2dot.perl [--organize-to-stacks] < temp.graph > temp.dot
dot -Tps temp.dot > temp.ps
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Using {\tt -{}-organize-to-stacks} makes nodes in the same stack appear in the same column (this slows down the rendering, off by default).



Caution: the input must contain the searchgraph of one sentence only.



\subsection{
\label{support-tools.texntoc13}Threshold Pruning of Phrase Table}


The phrase table trained by Moses contains by default all phrase pairs encountered in the parallel training corpus. This often includes 100,000 different translations for the word "the" or the comma ",". These may clog up various processing steps down the road, so it is helpful to prune the phrase table to the reasonable choices. 



Threshold pruning is currently implemented at two different stages: You may filter the entire phrase table file, or use threshold pruning as an additional filtering criterion when filtering the phrase table for a given test set. In either case, phrase pairs are thrown out when their phrase translation probability p(e$|$f) falls below a specified threshold. A safe number for this threshold may be 0.0001, in the sense that it hardly changes any phrase translation while ridding the table of a lot of junk.



\subsubsection*{
\label{support-tools.texntoc14}Pruning the full phrase table file}


The script {\tt scripts/training/threshold-filter.perl} operates on any phrase table file:






\begin{SaveVerbatim}{myverb}
cat PHRASE_TABLE | \ 
threshold-filter.perl 0.0001 > PHRASE_TABLE.reduced
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If the phrase table is zipped, then:






\begin{SaveVerbatim}{myverb}
zcat PHRASE_TABLE.gz | \ 
threshold-filter.perl 0.0001 | \ 
gzip - > PHRASE_TABLE.reduced.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

While this often does not remove much of the phrase table (which contains to a large part singleton phrase pairs with p(e$|$f)=1), it may nevertheless be helpful to also reduce the reordering model. This can be done with a second script:






\begin{SaveVerbatim}{myverb}
cat REORDERING_TABLE | \ 
remove-orphan-phrase-pairs-from-reordering-table.perl PHRASE_TABLE \ 
> REORDERING_TABLE.pruned
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Again, this also works for zipped files:






\begin{SaveVerbatim}{myverb}
zcat REORDERING_TABLE.gz | \ 
remove-orphan-phrase-pairs-from-reordering-table.perl PHRASE_TABLE | \  
gzip - > REORDERING_TABLE.pruned.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
\label{support-tools.texntoc15}Pruning during test/tuning set filtering}


In the typical experimental setup, the phrase table is filtered for a tuning or test set using the {\tt } script. During this process, we can also remove low-probability phrase pairs. This can be done simply by adding the switch {\tt -MinScore}, which takes a specification of the following form:






\begin{SaveVerbatim}{myverb}
filter-model-given-input.pl [...]  \ 
-MinScore FIELD1:THRESHOLD2[,FIELD2:THRESHOLD2[,FIELD3:THRESHOLD3]]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

where {\tt FIELDn} is the position of the score (typically 2 for the direct phrase probability p(e$|$f), or 0 for the indirect phrase probability p(f$|$e)) and {\tt THRESHOLD} the maximum probability allowed.







\begin{center}\textit{\small Subsection last  modified on February 06, 2016, at 10:11 PM}\end{center}
