



\section{
\label{training-intro.tex}
Training}


\subsection{
\label{training-intro.texntoc1}Training process}


We will start with an overview of the training process. This should give a feel for what is going on and what files are produced. In the following, we will go into more details  of the options of the training process and additional tools.



The training process takes place in nine steps, all of them executed by the script

\begin{SaveVerbatim}{myverb}
train-model.perl
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The nine steps are

\begin{compactenum}\item Prepare data (45 minutes)

\item Run GIZA++ (16 hours)

\item Align words (2:30 hours)

\item Get lexical translation table (30 minutes)

\item Extract phrases (10 minutes)

\item Score phrases (1:15 hours)

\item Build lexicalized reordering model (1 hour)

\item Build generation models 

\item Create configuration file (1 second)

\end{compactenum}
If you are running on a machine with multiple processors, some of these steps can be considerably sped up with the following option:

\begin{SaveVerbatim}{myverb}
--parallel
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The run times mentioned in the steps refer to a recent training run on the 751'000 sentence, 16 million word German-English Europarl corpus, on a 3GHz Linux machine.



If you wish to experiment with translation in both directions, step 1 and 2 can be reused, starting from step 3 the contents of the model directory get direction-dependent. In other words run steps 1 and 2, then make a copy of the whole experiment directory and continue two trainings from step 3.



\subsection{
\label{training-intro.texntoc2}Running the training script}


For an standard phrase model, you will typically run the training script as follows.  



Run the training script:






\begin{SaveVerbatim}{myverb}
train-model.perl -root-dir . --corpus corpus/euro --f de --e en
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

There should be two files in the {\em corpus/} directory called {\em euro.de} and {\em euro.en}.  These files should be sentence-aligned halfs of the parallel corpus.  {\em euro.de} should contain the German sentences, and {\em euro.en} should contain the corresponding English sentences.



More on the training parameters (Section \ref{training-reference.tex}) at the end of this manual. For corpus preparation, see the section on how to prepare training data (Section \ref{training-prepare.tex}).







\begin{center}\textit{\small Subsection last  modified on May 04, 2010, at 10:05 PM}\end{center}
