



\section{
\label{optimize.tex}
Optimizing Moses}


\subsection{
\label{optimize.texntoc1}Multi-threaded Moses}


Moses supports multi-threaded operation, enabling faster decoding on multi-core machines. The current limitations of multi-threaded Moses are:

\begin{compactenum}\item irstlm is not supported, since it uses a non-threadsafe cache

\item lattice input may not work - this has not been tested

\item increasing the verbosity of Moses will probably cause multi-threaded Moses to crash

\item Decoding speed will flatten out after about 16 threads. For more scalable speed with many threads, use Moses2

\end{compactenum}
Multi-threaded Moses is now built by default. If you omit the {\tt -threads} argument, then Moses will use a single worker thread, and a thread to read the input stream. Using the argument {\tt -threads n} specifies a pool of {\tt n} threads, and {\tt -threads all} will use all the cores on the machine.



\subsection{
\label{optimize.texntoc2}How much memory do I need during decoding?}


The single-most important thing you need to run Moses fast is {\bf MEMORY}. Lots of {\bf MEMORY}. (For example, the Edinburgh group have servers with 144GB of RAM). The rest of this section is just details of how to make the training and decoding run fast.



Calculate total file size of the binary phrase tables, binary language models and binary reordering models.



For example,

\begin{SaveVerbatim}{myverb}
% ll -h phrase-table.0-0.1.1.binphr.*
-rw-r--r-- 1 s0565741 users 157K 2012-06-13 12:41 phrase-table.0-0.1.1.binphr.idx
-rw-r--r-- 1 s0565741 users 5.4M 2012-06-13 12:41 phrase-table.0-0.1.1.binphr.srctree
-rw-r--r-- 1 s0565741 users 282K 2012-06-13 12:41 phrase-table.0-0.1.1.binphr.srcvoc
-rw-r--r-- 1 s0565741 users 1.1G 2012-06-13 12:41 phrase-table.0-0.1.1.binphr.tgtdata
-rw-r--r-- 1 s0565741 users 1.7M 2012-06-13 12:41 phrase-table.0-0.1.1.binphr.tgtvoc
% ll -h reordering-table.1.wbe-msd-bidirectional-fe.binlexr.*
-rw-r--r-- 1 s0565741 users 157K 2012-06-13 13:36 reordering-table.1.wbe-msd-bidirectional-fe.binlexr.idx
-rw-r--r-- 1 s0565741 users 1.1G 2012-06-13 13:36 reordering-table.1.wbe-msd-bidirectional-fe.binlexr.srctree
-rw-r--r-- 1 s0565741 users 1.1G 2012-06-13 13:36 reordering-table.1.wbe-msd-bidirectional-fe.binlexr.tgtdata
-rw-r--r-- 1 s0565741 users 282K 2012-06-13 13:36 reordering-table.1.wbe-msd-bidirectional-fe.binlexr.voc0
-rw-r--r-- 1 s0565741 users 1.7M 2012-06-13 13:36 reordering-table.1.wbe-msd-bidirectional-fe.binlexr.voc1
% ll -h interpolated-binlm.1 
-rw-r--r-- 1 s0565741 users 28G 2012-06-15 11:07 interpolated-binlm.1
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

The total size of these files is approx. 31GB. Therefore, a translation system using these models requires 31GB (+ roughly 500MB) of memory to run fast.   



\subsubsection*{
\label{optimize.texntoc3}I've got this much memory but it's still slow. Why?}


Run this:

\begin{SaveVerbatim}{myverb}
cat phrase-table.0-0.1.1.binphr.* > /dev/null
cat reordering-table.1.wbe-msd-bidirectional-fe.binlexr.* > /dev/null
cat interpolated-binlm.1 > /dev/null
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This forces the operating system to cache the binary models in memory, minimizing pages faults while the decoder is running. Other memory-intensive processes on the computer should not be running, otherwise the file-system cache may be reduced.



\subsubsection*{
\label{optimize.texntoc4}Use huge pages}


Moses does a lot of random lookups.  If you're running Linux, check that transparent huge pages\footnote{\sf https://lwn.net/Articles/423584/} are enabled.  If

\begin{SaveVerbatim}{myverb}
cat /sys/kernel/mm/transparent_hugepage/enabled
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

responds with

\begin{SaveVerbatim}{myverb}
[always] madvise never
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

then transparent huge pages are enabled.  



On some RedHat/Centos systems, the file is {\tt /sys/kernel/mm/redhat\_transparent\_hugepage/enabled} and {\tt madvise} will not appear.  If neither file exists, upgrade the kernel to at least 2.6.38 and compile with {\tt CONFIG\_SPARSEMEM\_VMEMMAP}.  If the file exists, but the square brackets are not around "always", then run

\begin{SaveVerbatim}{myverb}
echo always > /sys/kernel/mm/transparent_hugepage/enabled
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

as root (NB: to use {\tt sudo}, quote the {\tt $>$} character).  This setting will not be preserved across reboots, so consider adding it to an init script.  



\subsubsection*{
\label{optimize.texntoc5}Use the compact phrase and reordering table representations to reduce memory usage by a factor of 10}


See the manual on binarized\footnote{\sf http://www2.statmt.org/moses/?n=Moses.AdvancedFeatures\#ntoc5} and compact\footnote{\sf http://www2.statmt.org/moses/?n=Moses.AdvancedFeatures\#ntoc6} phrase table for a description how to compact your phrase tables. All the things said above for the standard binary phrase table are also true for the compact versions. The principle is the same, the total size of the binary files determines your memory usage, but since the combined size of the compact phrase table and the compact reordering model maybe up to 10 to 12 times smaller than with the original binary implementations, you will save exactly this much memory. You can also use the {\tt -{}-minphr-memory} and {\tt -{}-minlexr-memory} options to load the tables into memory at Moses start-up instead of doing the above mentioned caching trick. This may take some time during warm-up, but may save a lot of time in the long term. If you are concerned for performance, see Junczys-Dowmunt (2012)\footnote{\sf http://ufal.mff.cuni.cz/pbml/98/art-junczys-dowmunt.pdf} for a comparison. There is virtually no overhead due to on-the-fly decompression on large-memory-systems and considerable speed-up on systems with limited memory.  



\subsection{
\label{optimize.texntoc6}How little memory can I get away with during decoding?}


The decoder can run on very little memory, about 200-300MB for phrase-based and 400-500MB for hierarchical decoding (according to Hieu). The decoder can run on an iPhone! And laptops.



However, it will be VERY slow, unless you have very small models or the models are on fast disks such as flash disks.



\subsection{
\label{optimize.texntoc7}Faster Training}


\subsubsection*{
\label{optimize.texntoc8}Parallel training}


When word aligning, using mgiza\footnote{\sf http://sourceforge.net/projects/mgizapp/} with multiple threads significantly speed up word alignment.



\paragraph*{
\label{optimize.texntoc9}MGIZA}


To use MGIZA with multiple threads in the Moses training script, add these arguments:

\begin{SaveVerbatim}{myverb}
.../train-model.perl -mgiza -mgiza-cpus 8 ....
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

To enable it in the EMS, add this to the [TRAINING] section

\begin{SaveVerbatim}{myverb}
[TRAINING]
training-options = "-mgiza -mgiza-cpus 8"
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\paragraph*{
\label{optimize.texntoc10}snt2cooc}


When running GIZA++ or MGIZA, the first stage involves running a program called

\begin{SaveVerbatim}{myverb}
snt2cooc
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This requires approximately 6GB+ for typical Europarl-size corpora (1.8 million sentences). For users without this amount of memory on their computers, an alternative version is included in MGIZA:

\begin{SaveVerbatim}{myverb}
snt2cooc.pl
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

To use this script, you must copy 2 files to the same place where {\tt snt2cooc} is run:

\begin{SaveVerbatim}{myverb}
snt2cooc.pl
snt2coocrmp
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Add this argument when running the Moses training script:

\begin{SaveVerbatim}{myverb}
.../train-model.perl -snt2cooc snt2cooc.pl
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
\label{optimize.texntoc11}Parallel Extraction}


Once word alignment is completed, the phrase table is created from the aligned parallel corpus. There are 2 main ways to speed up this part of the training process.



Firstly, the training corpus and alignment can be split and phrase pairs from each part can be extracted simultaneously. This can be done by simply using the argument {\tt -cores}, e.g.,

\begin{SaveVerbatim}{myverb}
.../train-model.perl -cores 4
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Secondly, the Unix {\tt sort} command is often executed during training. It is essential to optimize this command to make use of the available disk and CPU. For example, recent versions of sort can take the following arguments

\begin{SaveVerbatim}{myverb}
sort  -S 10G --batch-size 253 --compress-program gzip --parallel 5
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The Moses training script names these arguments

\begin{SaveVerbatim}{myverb}
.../train-model.perl  -sort-buffer-size 10G -sort-batch-size 253 \
-sort-compress gzip -sort-parallel 5
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

You should set these arguments. However, DO NOT just blindly copy the above settings, they must be tuned to the particular computer you are running on. The most important issues are:

\begin{compactenum}\item you must make sure the version of sort on your machine supports the arguments you specify, otherwise the script will crash. The {\tt -{}-parallel}, {\tt -{}-compress-program}, and {\tt -{}-batch-size} arguments have only recently been added to the sort command.

\item make sure you have enough memory when setting {\tt -sort-buffer-size}. In particular, you should take into account other programs running on the computer. Also, two or three simultaneous sort program will run (one to sort the {\tt extract} file, one to sort {\tt extract.inv}, one to sort {\tt extract.o}). If there is not enough memory because you've set {\tt sort-buffer-size} too high, your entire computer will likely crash.

\item the maximum number for the {\tt -{}-batch-size} argument is OS-dependent. For example, it is 1024 on Linux, 253 on old Mac OSX, 2557 on new OSX.

\item on Mac OSX, using {\tt -{}-compress-program} can occasionally result in the following timeout errors. 

\end{compactenum}\begin{SaveVerbatim}{myverb}
gsort: couldn't create process for gzip -d: Operation timed out
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{optimize.texntoc12}Training Summary}


In summary, to maximize speed on a large server with many cores and up-to-date software, add this to your training script:

\begin{SaveVerbatim}{myverb}
.../train-model.perl -mgiza -mgiza-cpus 8 -cores 10 \
-parallel -sort-buffer-size 10G -sort-batch-size 253 \
-sort-compress gzip -sort-parallel 10
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

To run on a laptop with limited memory 

\begin{SaveVerbatim}{myverb}
.../train-model.perl -mgiza -mgiza-cpus 2 -snt2cooc snt2cooc.pl \
-parallel -sort-batch-size 253 -sort-compress gzip
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

In the EMS, for large servers, this can be done by adding:

\begin{SaveVerbatim}{myverb}
[TRAINING]
script = $moses-script-dir/training/train-model.perl
training-options = "-mgiza -mgiza-cpus 8 -cores 10 \
-parallel -sort-buffer-size 10G -sort-batch-size 253 \
-sort-compress gzip -sort-parallel 10"
parallel = yes
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

For servers with older OSes, and therefore older sort commands:

\begin{SaveVerbatim}{myverb}
[TRAINING]
script = $moses-script-dir/training/train-model.perl
training-options = "-mgiza -mgiza-cpus 8 -cores 10 -parallel"
parallel = yes
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

For laptops with limited memory:

\begin{SaveVerbatim}{myverb}
[TRAINING]
script = $moses-script-dir/training/train-model.perl
training-options = "-mgiza -mgiza-cpus 2 -snt2cooc snt2cooc.pl \
-parallel -sort-batch-size 253 -sort-compress gzip"
parallel = yes
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{optimize.texntoc13}Language Model}


Convert your language model to binary format.  This reduces loading time and provides more control.  



\subsubsection*{
\label{optimize.texntoc14}Building a KenLM binary file}


See the KenLM web site\footnote{\sf http://kheafield.com/code/kenlm/benchmark/} for the time-memory tradeoff presented by the KenLM data structures.  Use {\tt bin/build\_binary} (found in the same directory as {\tt moses} and {\tt moses\_chart}) to convert ARPA files to the binary format.  You can preview memory consumption with:

\begin{SaveVerbatim}{myverb}
bin/build_binary file.arpa
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This preview includes only the language model's memory usage, which is in addition to the phrase table etc.  For speed, use the default probing data structure.   

\begin{SaveVerbatim}{myverb}
bin/build_binary file.arpa file.binlm
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

To save memory, change to the trie data structure

\begin{SaveVerbatim}{myverb}
bin/build_binary trie file.arpa file.binlm
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

To further losslessly compress the trie ("chop" in the benchmarks), use {\tt -a 64} which will compress pointers to a depth of up to 64 bits.  

\begin{SaveVerbatim}{myverb}
bin/build_binary -a 64 trie file.arpa file.binlm
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Note that you can also make this parameter smaller which will go faster but use more memory.  Quantization will make the trie smaller at the expense of accuracy.  You can choose any number of bits from 2 to 25, for example 10:

\begin{SaveVerbatim}{myverb}
bin/build_binary -a 64 -q 10 trie file.arpa file.binlm
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Note that quantization can be used independently of -a.  



\subsubsection*{
\label{optimize.texntoc15}Loading on-demand}


By default, language models fully load into memory at the beginning.  If you are short on memory, you can use on-demand language model loading.  The language model must be converted to binary format in advance and should be placed on LOCAL DISK, preferably SSD.  For KenLM, you should use the trie data structure, not the probing data structure.  



If the LM for binarized using IRSTLM, append .mm to the file name and change the ini file to reflect this. Eg. change

\begin{SaveVerbatim}{myverb}
[feature]
IRSTLM .... path=file.lm
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

to

\begin{SaveVerbatim}{myverb}
[feature]
IRSTLM .... path=file.lm.mm
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If the LM was binarized using KenLM, add the argument lazyken=true. Eg. from

\begin{SaveVerbatim}{myverb}
[feature]
KENLM ....
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

to

\begin{SaveVerbatim}{myverb}
[feature]
KENLM .... lazyken=true
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{optimize.texntoc16}Suffix array}


Suffix arrays store the entire parallel corpora and word alignment information in memory, instead of the phrase table. The parallel corpora and alignment file is often much smaller than the phrase table. For example, for the Europarl German-English (gzipped files):

\begin{SaveVerbatim}{myverb}
de = 94MB
en = 84MB
alignment = 57MB

phrase-based = 2.0GB
hierarchical = 16.0GB
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Therefore, it is more memory efficient to store the corpus in memory, rather than the entire phrase-table. This is usually structured as a suffix array to enable fast extraction of translations. 



Translations are extracted as needed, usually per input test set, or per input sentence.



Moses support two different implementations of suffix arrays, one for phrase-based models\footnote{\sf http://www2.statmt.org/moses/?n=Moses.AdvancedFeatures\#ntoc35}, [[one for hierarchical models -$>$ AdvancedFeatures\#ntoc43
]]. 



\subsection{
\label{optimize.texntoc17}Cube Pruning}


Cube pruning limits the number of hypotheses created for each stack (or chart cell in chart decoding). It is essential for chart decoding (otherwise decoding will take a VERY long time) and an option in phrase-based decoding.



In the phrase-based decoder, add:

\begin{SaveVerbatim}{myverb}
[search-algorithm]
1
[cube-pruning-pop-limit]
500
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

There is a speed-quality tradeoff, lower pop limit means less work for the decoder, so faster decoding but less accurate translation.



\subsection{
\label{optimize.texntoc18}Minimizing memory during training}


TODO: MGIZA with reduced memory {\tt sntcoc}



\subsection{
\label{optimize.texntoc19}Minimizing memory during decoding}


The biggest consumer of memory during decoding are typically the models. Here are some links on how to reduce the size of each.



Language model: 

\begin{SaveVerbatim}{myverb}
* use KenLM with trie data structure Moses.Optimize#ntoc14\footnote{\sf http://www2.statmt.org/moses/?n=Moses.Optimize#ntoc14}
* use on-demand loading Moses.Optimize#ntoc15\footnote{\sf http://www2.statmt.org/moses/?n=Moses.Optimize#ntoc15}
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

Translation model: 

\begin{SaveVerbatim}{myverb}
* use phrase table pruning Advanced.RuleTables#ntoc5\footnote{\sf http://www2.statmt.org/moses/?n=Advanced.RuleTables#ntoc5}
* use a compact phrase table http://www.statmt.org/moses/?n=Advanced.RuleTables#ntoc3
* filter the translation model given the text you want to translate Moses.SupportTools#ntoc3\footnote{\sf http://www2.statmt.org/moses/?n=Moses.SupportTools#ntoc3}
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

Reordering model:

\begin{SaveVerbatim}{myverb}
* similar techniques than for translation models are possible: pruning Advanced.RuleTables#ntoc3\footnote{\sf http://www2.statmt.org/moses/?n=Advanced.RuleTables#ntoc3}, compact tables Advanced.RuleTables#ntoc4\footnote{\sf http://www2.statmt.org/moses/?n=Advanced.RuleTables#ntoc4}, and filtering Moses.SupportTools#ntoc3\footnote{\sf http://www2.statmt.org/moses/?n=Moses.SupportTools#ntoc3}.
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
\label{optimize.texntoc20}Compile-time options}


These options can be added to the bjam command line, trading generality for performance.  



You should do a full rebuild with {\tt -a} when changing the values of most of these options.



Don't use factors?  Add 

\begin{SaveVerbatim}{myverb}
--max-factors=1
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Tailor KenLM's maximum order to only what you need.  If your highest-order language model has order 5, add

\begin{SaveVerbatim}{myverb}
--kenlm-max-order=5
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Turn debug symbols off for speed and a little more memory.  

\begin{SaveVerbatim}{myverb}
debug-symbols=off
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

But don't expect support from the mailing list until you rerun with debug symbols on!  



Don't care about debug messages?

\begin{SaveVerbatim}{myverb}
--notrace
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Download {\tt tcmalloc}\footnote{\sf https://code.google.com/p/gperftools} and see {\tt BUILD-INSTRUCTIONS.txt} in Moses for installation instructions.  {\tt bjam} will automatically detect tcmalloc's presence and link against it for multi-threaded builds.  



Install Boost and {\tt zlib} static libraries.  Then link statically:

\begin{SaveVerbatim}{myverb}
--static
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This may mean you have to install Boost and {\tt zlib} yourself.  



Running single-threaded?  Add {\tt threading=single}.



Using hierarchical or string-to-tree models, but none with source syntax?

\begin{SaveVerbatim}{myverb}
--unlabelled-source
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{optimize.texntoc21}Phrase-table types}


Moses has multiple phrase table implementations. The one that suits you best depends on the model you're using (phrase-based or hierarchical/syntax), and how much memory your server has.



Here is a complete list of the types:



{\bf Memory} - this read in the phrase table into memory. For phrase-based model and chart decoding. Note that this is much faster than Binary and OnDisk phrase table format, but it uses a lot of RAM.  



{\bf Binary} - a phrase table is converted into a 'database'. Only the
translations which are required are loaded into memory. Therefore,
requiring less memory, but potentially slower to run. For phrase-based model



{\bf OnDisk} - reimplementation of Binary for chart decoding.



{\bf SuffixArray} - stores the parallel training data and word alignment in
memory, instead of the phrase table. Extraction is done on the fly. Also
have a feature where you can add parallel data while the decoder is
running ('Dynamic Suffix Array'). For Phrase-based models. See Levenberg et al., (2010)\footnote{\sf http://homepages.inf.ed.ac.uk/miles/papers/naacl10b.pdf}.



{\bf ALSuffixArray} - Suffix array for hierarchical models. See Lopez (2008)\footnote{\sf http://www.cs.jhu.edu/\textasciitilde{}alopez/talks/mtm2008-lopez.pdf}.



{\bf FuzzyMatch} - Implementation of Koehn and Senellart (2010)\footnote{\sf http://homepages.inf.ed.ac.uk/pkoehn/publications/tm-smt-amta2010.pdf}.



{\bf Hiero} - like SCFG, but translation rules are in standard Hiero-style format



{\bf Compact} - for phrase-based model. See Junczys-Dowmunt (2012)\footnote{\sf http://ufal.mff.cuni.cz/pbml/98/art-junczys-dowmunt.pdf}.







\begin{center}\textit{\small Subsection last  modified on December 15, 2016, at 01:50 PM}\end{center}
