



\section{
\label{get-started.tex}
Getting Started with Moses}


This section will show you how to install and build Moses, and how to use Moses to translate with some simple models. If you experience problems, then please check the support\footnote{\sf http://www2.statmt.org/moses/?n=Moses.MailingLists} page. If you do not want to build Moses from source, then there are packages\footnote{\sf http://www2.statmt.org/moses/?n=Moses.Packages} available for Windows and popular Linux distributions.



\subsection{
Compiling Moses with bjam}


To compile with bare minimum of features:

\begin{SaveVerbatim}{myverb}
./bjam -j4
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If you have compiled boost manually, then tell bjam where it is:

\begin{SaveVerbatim}{myverb}
./bjam --with-boost=~/workspace/temp/boost_1_64_0 -j8
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If you have compiled the cmph library manually:

\begin{SaveVerbatim}{myverb}
./bjam --with-cmph=/Users/hieu/workspace/cmph-2.0
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If you have compiled the xmlrpc-c library manually:

\begin{SaveVerbatim}{myverb}
./bjam --with-xmlrpc-c=/Users/hieu/workspace/xmlrpc-c/xmlrpc-c-1.33.17
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If you have compiled the xmlrpc-c library manually:

\begin{SaveVerbatim}{myverb}
./bjam --with-irstlm=/Users/hieu/workspace/irstlm/irstlm-5.80.08/trunk 
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This is the exact command I (Hieu) used on Linux:

\begin{SaveVerbatim}{myverb}
./bjam --with-boost=/home/s0565741/workspace/boost/boost_1_57_0 --with-cmph=/home/s0565741/workspace/cmph-2.0 --with-irstlm=/home/s0565741/workspace/irstlm-code --with-xmlrpc-c=/home/s0565741/workspace/xmlrpc-c/xmlrpc-c-1.33.17 -j12
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
Manually installing Boost}


Boost 1.48 has a serious bug which breaks Moses compilation. Unfortunately, some Linux distributions (eg. Ubuntu 12.04) have broken versions of the Boost library. In these cases, you must download and compile Boost yourself. 



This is the exact commands I (Hieu) use to compile boost:

\begin{SaveVerbatim}{myverb}
wget https://dl.bintray.com/boostorg/release/1.64.0/source/boost_1_64_0.tar.gz
tar zxvf boost_1_64_0.tar.gz 
cd boost_1_64_0/
./bootstrap.sh 
./b2 -j4 --prefix=$PWD --libdir=$PWD/lib64 --layout=system link=static install || echo FAILURE
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

This create library file in the directory lib64, NOT in the system directory. Therefore, you don't need to be system admin/root to run this. However, you will need to tell moses where to find boost, which is explained below



Once boost is installed, you can then compile Moses. However, you must tell Moses where boost is with the -{}-with-boost flag. This is the exact commands I use to compile Moses:

\begin{SaveVerbatim}{myverb}
./bjam --with-boost=~/workspace/temp/boost_1_64_0 -j4
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Other software to install}


\subsubsection*{
Word Alignment}


Moses requires a word alignment tool, such as giza++\footnote{\sf http://code.google.com/p/giza-pp/}, mgiza\footnote{\sf https://github.com/moses-smt/mgiza}, or Fast Align\footnote{\sf https://github.com/clab/fast\_align/blob/master/README.md}. 



I (Hieu) use MGIZA because it is multi-threaded and give general good result, however, I've also heard good things about Fast Align. You can find  instructions to compile them here\footnote{\sf http://www.statmt.org/moses/?n=Moses.ExternalTools\#ntoc3}.



\subsubsection*{
Language Model Creation}


Moses includes the KenLM language model creation program, lmplz\footnote{\sf http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel\#ntoc19}.  



You can also create language models with IRSTLM\footnote{\sf http://sourceforge.net/projects/irstlm/} and SRILM\footnote{\sf http://www.speech.sri.com/projects/srilm/download.html}.  Please read this\footnote{\sf http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel\#ntoc4} if you want to compile IRSTLM.  Language model toolkits perform two main tasks: training and querying.  You can train a language model with any of them, produce an ARPA file, and query with a different one.  To train a model, just call the relevant script.  



If you want to use SRILM or IRSTLM to query the language model, then they need to be linked with Moses.  For IRSTLM, you first need to compile IRSTLM then use the -{}-with-irstlm switch to compile Moses with IRSTLM. This is the exact command I used:

\begin{SaveVerbatim}{myverb}
./bjam --with-irstlm=/home/s0565741/workspace/temp/irstlm-5.80.03 -j4
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Personally, I only use IRSTLM as a query tool in this way if the LM n-gram order is over 7. In most situation, I use KenLM because KenLM is multi-threaded and faster.



\subsection{
Platforms}


The primary development platform for Moses is Linux, and this is the recommended platform since you will find it easier to get support for it. However Moses does work on other platforms:



\subsection{
Linux Installation}


\subsubsection*{
Debian}


Install the following packages using the command

\begin{SaveVerbatim}{myverb}
su
apt-get install [package name]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Packages:

\begin{SaveVerbatim}{myverb}
git
subversion
make
libtool
gcc
g++
libboost-dev
tcl-dev
tk-dev
zlib1g-dev
libbz2-dev
python-dev
libicu-dev (Debian)
libunistring-dev (Debian)
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
Ubuntu}


Install the following packages using the command

\begin{SaveVerbatim}{myverb}
sudo apt-get install [package name]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Packages:

\begin{SaveVerbatim}{myverb}
g++ 
git 
subversion
automake
libtool
zlib1g-dev
libicu-dev
libboost-all-dev
libbz2-dev
liblzma-dev
python-dev
graphviz
imagemagick
make
cmake
libgoogle-perftools-dev (for tcmalloc)
autoconf
doxygen
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
Fedora / Redhat / CentOS / Scientific Linux}


Install the following packages using the command

\begin{SaveVerbatim}{myverb}
su
yum install [package name]
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Packages:

\begin{SaveVerbatim}{myverb}
git
subversion
make
automake
cmake
libtool
gcc-c++
zlib-devel
python-devel
bzip2-devel
boost-devel
ImageMagick
cpan
expat-devel
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

In addition, you have to install some perl packages:

\begin{SaveVerbatim}{myverb}
cpan XML::Twig
cpan Sort::Naturally
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
OSX Installation}


Mac OSX is widely used by Moses developers and everything should run fine. Installation is the same as for Linux.



Mac OSX out-of-the-box doesn't have many programs that are critical to Moses, or different version of standard GNU programs. For example, {\tt split}, {\tt sort}, {\tt zcat} are incompatible BSD-versions rather than GNU versions.



Therefore, Moses has been tested with Mac OSX with {\bf Mac Ports}. Make sure you have this installed on your machine. Success has also been reported with {\tt brew} installation. Do note, however, that you will need to install {\tt xmlrpc-c} independently, and then compile with {\tt bjam} using the {\tt -{}-with-xmlrpc-c=/usr/local} flag (where /usr/local/ is the default location of the xmlrpc-c installation).



Recent versions of OSX have clang C/C++ compiler, rather than gcc. When compiling with bjam, you must add the following:

\begin{SaveVerbatim}{myverb}
./bjam toolset=clang
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This is the exact command I (Hieu) use on OSX Yosemite:

\begin{SaveVerbatim}{myverb}
./bjam --with-boost=/Users/hieu/workspace/boost/boost_1_59_0.clang/ --with-cmph=/Users/hieu/workspace/cmph-2.0 --with-xmlrpc-c=/Users/hieu/workspace/xmlrpc-c/xmlrpc-c-1.33.17 --with-irstlm=/Users/hieu/workspace/irstlm/irstlm-5.80.08/trunk --with-mm --with-probing-pt -j5 toolset=clang -q -d2
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

You also need to add this argument when manually compiling boost. This is the exact command I use:

\begin{SaveVerbatim}{myverb}
./b2 -j8 --prefix=$PWD --libdir=$PWD/lib64 --layout=system link=static toolset=clang  install || echo FAILURE
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Windows Installation}


Moses can run on Windows 10 with Ubuntu 16.04 subsystem, available within windows programs' feature tab. More information here:

\begin{SaveVerbatim}{myverb}
https://docs.microsoft.com/en-us/windows/wsl/install-win10
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Thereafter, installation is exactly the same as for Ubuntu. 



\subsection{
Run Moses for the first time}


Download the sample models and extract them into your working directory:






\begin{SaveVerbatim}{myverb}
cd ~/mosesdecoder
wget http://www.statmt.org/moses/download/sample-models.tgz
tar xzf sample-models.tgz
cd sample-models
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Run the decoder

\begin{SaveVerbatim}{myverb}
cd ~/mosesdecoder/sample-models
~/mosesdecoder/bin/moses -f phrase-model/moses.ini < phrase-model/in > out
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

If everything worked out right, this should translate the sentence "das ist ein kleines haus" (in the file {\tt in}) as "this is a small house" (in the file {\tt out}).



Note that the configuration file {\tt moses.ini} in each directory is set to use the KenLM language model toolkit by default.  If you prefer to use IRSTLM\footnote{\sf http://hlt.fbk.eu/en/irstlm}, then edit the language model entry in {\tt moses.ini}, replacing {\tt KENLM} with {\tt IRSTLM}. You will also have to compile with {\tt ./bjam -{}-with-irstlm}, adding the full path of your IRSTLM installation.



Moses also supports SRILM and RandLM language models. See here\footnote{\sf http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel\#ntoc1} for more details.



\label{get-started.texchart}


\subsection{
Chart Decoder}


The chart decoder is part of the same executable as of version 3.0.



You can run the chart demos from the sample-models directory as follows

\begin{SaveVerbatim}{myverb}
~/mosesdecoder/bin/moses -f string-to-tree/moses.ini < string-to-tree/in > out.stt
~/mosesdecoder/bin/moses -f tree-to-tree/moses.ini < tree-to-tree/in.xml > out.ttt
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The expected result of the string-to-tree demo is

\begin{SaveVerbatim}{myverb}
this is a small house
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Next Steps}


Why not try to build a Baseline (Section \ref{baseline.tex}) translation system with freely available data?



\subsection{
bjam options}


\label{get-started.texbjam}
This is a list of options to bjam.  On a system with Boost installed in a standard path, none should be required, but you may want additional functionality or control.  



\subsubsection*{
Optional packages}


\paragraph*{
Language models }


In addition to KenLM and ORLM (which are always compiled):

\begin{description}\item[-{}-with-irstlm=/path/to/irstlm]Path to IRSTLM installation  
\item[-{}-with-randlm=/path/to/randlm]Path to RandLM installation
\item[-{}-with-nplm=/path/to/nplm]Path to NPLM installation
\item[-{}-with-srilm=/path/to/srilm]Path to SRILM installation.
\end{description}
If your SRILM install is non-standard, use these options:

\begin{description}\item[-{}-with-srilm-dynamic]Link against srilm.so.
\item[-{}-with-srilm-arch=arch]Override the arch setting given by /path/to/srilm/sbin/machine-type
\end{description}
\paragraph*{
Other packages}

\begin{description}\item[-{}-with-boost=/path/to/boost]If Boost is in a non-standard location, specify it here.  This directory is expected to contain include and lib or lib64.  
\item[-{}-with-xmlrpc-c=/path/to/xmlrpc-c]Specify a non-standard libxmlrpc-c installation path.  Used by Moses server.
\item[-{}-with-cmph=/path/to/cmph]Path where CMPH is installed.  Used by the compact phrase table and compact lexical reordering table.   
\item[-{}-without-tcmalloc]Disable thread-caching malloc.  
\item[-{}-with-regtest=/path/to/moses-regression-tests]Run the regression tests using data from this directory.  Tests can be downloaded from {\tt {\sf https://github.com/moses-smt/moses-regression-tests}}.
\end{description}
\subsubsection*{
Installation}

\begin{description}\item[-{}-prefix=/path/to/prefix]sets the install prefix [default is source root].
\item[-{}-bindir=/path/to/prefix/bin]sets the bin directory [default is {\tt PREFIX/bin}]
\item[-{}-libdir=/path/to/prefix/lib]sets the lib directory [default is {\tt PREFIX/lib}]
\item[-{}-includedir=/path/to/prefix/include]installs headers.  Does not install if missing.  No argument defaults to {\tt PREFIX/include} .
\item[-{}-install-scripts=/path/to/scripts]copies scripts into a directory. Does not install if missing.  No argument defaults to {\tt PREFIX/scripts} .
\item[-{}-git] appends the git revision to the prefix directory.
\end{description}
\subsubsection*{
Build Options}


By default, the build is multi-threaded, optimized, and statically linked.  






\begin{description}\item[threading=single$|$multi]controls threading (default multi)
\item[variant=release$|$debug$|$profile]builds optimized (default), for debug, or for profiling
\item[link=static$|$shared]controls preferred linking (default static)
\item[-{}-static]forces static linking (the default will fall back to shared)
\item[debug-symbols=on$|$off]include (default) or exclude debugging information also known as -g
\item[-{}-notrace]compiles without TRACE macros
\item[-{}-enable-boost-pool]uses Boost pools for the memory SCFG table
\item[-{}-enable-mpi]switch on mpi (used for MIRA - one of the tuning algorithms)
\item[-{}-without-libsegfault]does not link with libSegFault
\item[-{}-max-kenlm-order]maximum ngram order that kenlm can process (default 6)
\item[-{}-max-factors]maximum number of factors (default 4)
\item[-{}-unlabelled-source] ignore source nonterminals (if you only use hierarchical or string-to-tree models without source syntax)
\end{description}
\subsubsection*{
Controlling the Build}

\begin{description}\item[-q]quit on the first error
\item[-a]to build from scratch
\item[-j\$NCPUS]to compile in parallel
\item[-{}-clean]to clean
\end{description}
\section{
Building with Eclipse}


There is a video showing you how to set up Moses with Eclipse.

\begin{SaveVerbatim}{myverb}
{\bf How to compile Moses with Eclipse\footnote{\sf https://vimeo.com/129306919}}
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Moses comes with Eclipse project files for some of the C++ executables. Currently, there are project files for 

\begin{SaveVerbatim}{myverb}
* moses (decoder)
* moses-cmd (decoder)
* extract
* extract-rules
* extract-ghkm
* server
* ...
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The Eclipse build is used primarily for development and debugging. It is not optimized and doesn't have many of the options available in the bjam build. 



The advantage of using Eclipse is that it offers code-completion, and a GUI debugging environment.



{\em NB.} The recent update of Mac OSX replaces g++ with clang. Eclipse doesn't yet fully function with clang. Therefore, you should not use the Eclipse build with any OSX version higher than 10.8 (Mountain Lion)



Follow these instructions to build with Eclipse:

\begin{SaveVerbatim}{myverb}
* Use the version of Eclipse for C++. Works (at least) with Eclipse Kepler and Luna.
* Get the Moses source code
git clone git@github.com:moses-smt/mosesdecoder.git
cd mosesdecoder
* Create a softlink to Boost (and optionally to XMLRPC-C lib if you want to compile the moses server) in the Moses root directory
eg. ln -s ~/workspace/boost_x_xx_x boost
* Create a new Eclipse workspace. The workspace {\em MUST} be in
contrib/other-builds/
Eclipse should now be running. 
* Import all the Moses Eclipse project into the workspace. 
File >> Import >> Existing Projects into Workspace >> Select root directory: contrib/other-builds/ >> Finish
* Compile all projects. 
Project >> Build All  
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

<del> !! Easy Setup on Ubuntu (on other linux systems, you'll need to install packages that provide gcc, make, git, automake, libtool) </del>



<del> \# Install required Ubuntu packages to build Moses and its dependencies:\\ </del>
<del>   {\tt sudo apt-get install build-essential git-core pkg-config automake libtool wget zlib1g-dev python-dev libbz2-dev}\\ </del>
<del>   For the regression tests, you'll also need \\ </del>
<del>   {\tt sudo apt-get install libsoap-lite-perl}\\ </del>
<del>   See below for additional packages that you'll need to actually run Moses (especially when you are using EMS). </del>
<del> \# Clone Moses from the repository and cd into the directory for building Moses\\ </del>
<del>   {\tt git clone {\sf https://github.com/moses-smt/mosesdecoder.git}} \\ </del>
<del>   {\tt cd mosesdecoder}  </del>
<del> \# Run the following to install a recent version of Boost (the default version on your system might be too old), as well as cmph (for CompactPT), irstlm (language model from FBK, required to pass the regression tests), and xmlrpc-c (for moses server). By default, these will be installed in ./opt in your working directory:\\ </del>
<del>   {\tt make -f contrib/Makefiles/install-dependencies.gmake} </del>
<del> \# To compile moses, run \\ </del>
<del>   {\tt ./compile.sh [additional options]} </del>



<del> !!! Popular additional bjam options (called from within {\tt ./compile.sh} and {\tt ./run-regtests.sh}): </del>
<del> * {\tt -{}-prefix=/destination/path -{}-install-scripts}\\ </del>
<del>   ... to install Moses somewhere else on your system  </del>



<del> * {\tt -{}-with-mm} \\ </del>
<del>   ...to enable suffix array-based phrase tables\footnote{\sf https://ufal.mff.cuni.cz/pbml/104/art-germann.pdf} </del>



<del> {\bf Note that you'll still need a word aligner; this is not built automatically} </del>



<del> !!! Running regression tests (Advanced; for Moses developers; normal users won't need this) </del>
<del> To compile and run the regression tests all in one go, run \\ </del>
<del> {\tt ./run-regtests.sh [additional options]} \\ </del>
<del> Regression testing is only of interest for people who are actively making changes in the Moses codebase. If you are just using Moses to run MT experiments, there's no point in running regression tests, unless you want to check that your current version of Moses is working as expected. However, you can also check your version against the daily regression tests here\footnote{\sf http://statmt.org/moses/cruise}.  </del>



<del> If you run your own regression tests, sometimes Moses will fail them even when everything is working correctly, because different compilers produce slightly different executables that might produce slightly different output because they make different kinds of rounding errors.   </del>












\begin{center}\textit{\small Subsection last  modified on January 17, 2020, at 12:44 AM}\end{center}
