



\section{
\label{training-prepare.tex}
Preparing Training Data}


Training data has to be provided sentence aligned (one sentence per line), in two files, one for the foreign sentences, one for the English sentences:






\begin{SaveVerbatim}{myverb}
>head -3 corpus/euro.*
==> corpus/euro.de <==
wiederaufnahme der sitzungsperiode
ich erklaere die am donnerstag , den 28. maerz 1996 unterbrochene 
sitzungsperiode des europaeischen parlaments fuer wiederaufgenommen .
begruessung

==> corpus/euro.en <==
resumption of the session
i declare resumed the session of the european parliament adjourned 
on thursday , 28 march 1996 .
welcome
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

A few other points have to be taken care of:

\begin{compactitem}
\item unix commands require the environment variable {\tt LC\_ALL=C}

\item one sentence per line, no empty lines

\item sentences longer than 100 words (and their corresponding translations) have to be eliminated (note that a shorter sentence length limit will speed up training

\item everything lowercased (use {\tt lowercase.perl})

\end{compactitem}

\subsection{
Training data for factored models}


You will have to provide training data in the format






\begin{SaveVerbatim}{myverb}
word0factor0|word0factor1|word0factor2 word1factor0|word1factor1|word1factor2 ...
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

instead of the un-factored






\begin{SaveVerbatim}{myverb}
word0 word1 word2
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Cleaning the corpus}


The script {\tt clean-corpus-n.perl} is small script that cleans up a parallel corpus, so it works well with the training script. 



It performs the following steps:

\begin{compactitem}
\item removes empty lines

\item removes redundant space characters

\item drops lines (and their corresponding lines), that are empty, too short, too long or violate the 9-1 sentence ratio limit of GIZA++

\end{compactitem}

The command syntax is:






\begin{SaveVerbatim}{myverb}
clean-corpus-n.perl CORPUS L1 L2 OUT MIN MAX
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

For example: {\tt clean-corpus-n.perl raw de en clean 1 50}
takes the corpus files {\tt raw.de} and {\tt raw.en},
deletes lines longer than 50, and creates the 
output files {\tt clean.de} and {\tt clean.en}.







\begin{center}\textit{\small Subsection last  modified on July 14, 2006, at 01:07 AM}\end{center}
