



\section{
\label{external-tools.tex}
External Tools}


A very active community is engaged in statistical machine translation research, which has produced a number of tools that may be useful for training a Moses system. Also, the more linguistically motivated models (factored model, syntax model) require tools to the linguistic annotation of corpora. 



In this section, we list some useful tools. If you know (or are the developer of) anything we missed here, please contact us and we can add it to the list. For more comprehensive listings of MT tools, refer to the following pages:






\begin{compactitem}
\item List of Free/Open-source MT Tools\footnote{\sf http://www.fosmt.org}, maintained by Mikel Forcada.

\item TAUS Tracker\footnote{\sf http://www.taustracker.com/}, a comprehensive list of Translation and Language Technology tools maintained by TAUS.

\end{compactitem}

\subsection{
\label{external-tools.texntoc1}Word Alignment Tools}


\label{external-tools.texberkeley}


\subsubsection*{
\label{external-tools.texntoc2}Berkeley Word Aligner}


The BerkeleyAligner\footnote{\sf http://nlp.cs.berkeley.edu/Main.html\#WordAligner} (available at Sourceforge\footnote{\sf http://code.google.com/p/berkeleyaligner/}) is a word alignment software package that implements recent innovations in unsupervised word alignment. It is implemented in Java and distributed in compiled format.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /my/installation/dir
cd /my/installation/dir
wget http://berkeleyaligner.googlecode.com/files/berkeleyaligner_unsupervised-2.1.tar.gz
tar xzf berkeleyaligner_unsupervised-2.1.tar.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Test:






\begin{SaveVerbatim}{myverb}
cd berkeleyaligner
chmod +x align 
./align example.conf
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\label{external-tools.texmgiza}


\subsubsection*{
\label{external-tools.texntoc3}Multi-threaded GIZA++}


MGIZA\footnote{\sf http://www.kyloo.net/software/doku.php} was developed by Qin Gao. It is an implementation of the popular GIZA++ word alignment toolkit to run multi-threaded on multi-core machines. Check the web site for more recent versions.



Installation:

\begin{SaveVerbatim}{myverb}
git clone https://github.com/moses-smt/mgiza.git
cd mgiza/mgizapp
cmake .
make
make install
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Compiling MGIZA requires the Boost library. If your Boost library are in non-system directory, use the script

\begin{SaveVerbatim}{myverb}
manual-compile/compile.sh
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

to compile MGIZA.



The MGIZA binary and the script {\tt merge\_alignment.py} need to be copied in you binary directory that Moses will look up for word alignment tools. This is the exact command I use to copy MGIZA to it final destination:

\begin{SaveVerbatim}{myverb}
export BINDIR=~/workspace/bin/training-tools
cp bin/* $BINDIR/mgizapp
cp scripts/merge_alignment.py $BINDIR
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

MGIZA works with the training script {\tt train-model.perl}. You indicate its use (opposed to regular GIZA++) with the switch {\tt -mgiza}. The switch {\tt -mgiza-cpus NUMBER} allows you to specify the number of CPUs.



\label{external-tools.texfastalign}


\subsubsection*{
\label{external-tools.texntoc4}Dyer et al.'s Fast Align}


The Fast Align\footnote{\sf https://github.com/clab/fast\_align/blob/master/README.md} is a comparable fast unsupervised word aligner that nevertheless gives comparable results to GIZA++. It's details are described in a NAACL 2013 paper\footnote{\sf http://www.ark.cs.cmu.edu/cdyer/fast\_valign.pdf}



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /my/installation/dir
cd /my/installation/dir
git clone https://github.com/clab/fast_align.git
cd fast_align
make
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\label{external-tools.texAnymalign}


\subsubsection*{
\label{external-tools.texntoc5}Anymalign}


Anymalign\footnote{\sf https://anymalign.limsi.fr/} is a multilingual sub-sentential aligner. It can extract lexical equivalences from sentence-aligned parallel corpora. Its main advantage over other similar tools is that it can align any number of languages simultaneously. The details are describe in Lardilleux and Lepage  (2009)\footnote{\sf http://www.aclweb.org/anthology/R09-1040}. To understand the algorithm, a pure python implementation can be found in minimalign.py\footnote{\sf https://anymalign.limsi.fr/minimalign.py} but it is advisable use the main implementation for realistic usage.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /your/installation/dir
cd /your/installation/dir
wget https://anymalign.limsi.fr/latest/anymalign2.5.zip
unzip anymalign2.5.zip
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{external-tools.texntoc6}Evaluation Metrics}


\subsubsection*{
\label{external-tools.texntoc7}Translation Error Rate (TER)}


Translation Error Rate\footnote{\sf http://www.cs.umd.edu/\textasciitilde{}snover/tercom/} is an error metric for machine translation that measures the number of edits required to change a system output into one of the references. It is implemented in Java.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /my/installation/dir
cd /my/installation/dir 
wget http://www.cs.umd.edu/~snover/tercom/tercom-0.7.25.tgz
tar xzf tercom-0.7.25.tgz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
\label{external-tools.texntoc8}METEOR}


METEOR\footnote{\sf http://www.cs.cmu.edu/\textasciitilde{}alavie/METEOR/} is a metric that includes stemmed and synonym matches when measuring the similarity between system output and human reference translations.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /my/installation/dir
cd /my/installation/dir 
wget http://www.cs.cmu.edu/~alavie/METEOR/install-meteor-1.0.sh
sh install-meteor-1.0.sh
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
\label{external-tools.texntoc9}RIBES}


RIBES\footnote{\sf http://www.kecl.ntt.co.jp/icl/lirg/ribes/} is a metric that word rank-based metric that compares the ratio of contiguous and dis-contiguous word pairs between the system output and human translations.



Installation:






\begin{SaveVerbatim}{myverb}
# First download from http://www.kecl.ntt.co.jp/icl/lirg/ribes/ 
# (need to accept to agree to the free license, so no direct URL)
tar -xvzf RIBES-1.03.1.tar.gz 
cd RIBES-1.03.1/
python RIBES.py --help
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
\label{external-tools.texntoc10}Part-of-Speech Taggers}


\label{external-tools.texmxpost}


\subsubsection*{
\label{external-tools.texntoc11}MXPOST (English)}


MXPOST was developed by Adwait Ratnaparkhi as part of his PhD thesis. It is a Java implementation of a maximum entropy model and distributed as compiled code. It can be trained for any language pair for with annotated POS data exists.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /your/installation/dir
cd /your/installation/dir
wget ftp://ftp.cis.upenn.edu/pub/adwait/jmx/jmx.tar.gz
tar xzf jmx.tar.gz 
echo '#!/usr/bin/env bash' > mxpost
echo 'export CLASSPATH=/your/installation/dir/mxpost.jar' >> mxpost
echo 'java -mx30m tagger.TestTagger /your/installation/dir/tagger.project' >> mxpost
chmod +x mxpost
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Test:






\begin{SaveVerbatim}{myverb}
echo 'This is a test .' | ./mxpost
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The script {\tt script/training/wrappers/make-factor-en-pos.mxpost.perl} is a wrapper script to create factors for a factored translation model. You have to adapt the definition of {\tt \$MXPOST} to point to your installation directory.



\subsubsection*{
\label{external-tools.texntoc12}TreeTagger (English, French, Spanish, German, Italian, Dutch, Bulgarian, Greek)}


TreeTagger\footnote{\sf http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/} is a tool for annotating text with part-of-speech and lemma information. 



Installation (Linux, check web site for other platforms):






\begin{SaveVerbatim}{myverb}
mkdir /my/installation/dir
cd /my/installation/dir
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/tree-tagger-linux-3.2.tar.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/tagger-scripts.tar.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/install-tagger.sh
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/english-par-linux-3.1.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/french-par-linux-3.2-utf8.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/spanish-par-linux-3.1.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/german-par-linux-3.2.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/italian-par-linux-3.2-utf8.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/dutch-par-linux-3.1.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/bulgarian-par-linux-3.1.bin.gz
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/greek-par-linux-3.2.bin.gz
sh install-tagger.sh
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The wrapper script {\tt scripts/training/wrapper/make-pos.tree-tagger.perl} creates part-of-speech factors using TreeTagger in the format expected by Moses. The command has the required parameters {\tt -tree-tagger DIR} to specify the location of your installation and {\tt -l LANGUAGE} to specify the two-letter code for the language ({\tt de}, {\tt fr}, ...). Optional parameters are {\tt -basic} to output only basic part-of-speech tags ({\tt VER} instead of {\tt VER:simp} -{}- not available for all languages), and {\tt -{}-stem} to output stems instead of part-of-speech tags.



Treetagger can also shallow parse the sentence, labelling it with chunk tags. See their website\footnote{\sf http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/DecisionTreeTagger.html} for details.



\label{external-tools.texfreeling}


\subsubsection*{
\label{external-tools.texntoc13}FreeLing}


FreeLing\footnote{\sf http://nlp.lsi.upc.edu/freeling/} is a set of a tokenizers, morpological analyzers, syntactic parsers. and other language tools for  Asturian, Catalan, English, Galician, Italian, Portuguese, Russian, Spanish, and Welsh. 



\subsection{
\label{external-tools.texntoc14}Syntactic Parsers}


\label{external-tools.texcollins}


\subsubsection*{
\label{external-tools.texntoc15}Collins (English)}


Michael Collins\footnote{\sf http://people.csail.mit.edu/mcollins/} developed the first statistical parser as part of his PhD thesis. It is implemented in C.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /your/installation/dir
cd /your/installation/dir
wget http://people.csail.mit.edu/mcollins/PARSER.tar.gz
tar xzf PARSER.tar.gz
cd COLLINS-PARSER/code
make
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Collins parser also requires the installation of MXPOST (Section \ref{external-tools.texmxpost}). A wrapper file to generate parse trees in the format required to train syntax models with Moses is provided in {\tt scrips/training/wrapper/parse-en-collins.perl}.



\label{external-tools.texbitpar}


\subsubsection*{
\label{external-tools.texntoc16}BitPar (German, English)}


Helmut Schmid developed BitPar\footnote{\sf http://www.cis.uni-muenchen.de/\textasciitilde{}schmid/tools/BitPar/}, a parser for highly ambiguous probabilistic context-free grammars (such as treebank grammars). BitPar uses bit-vector operations to speed up the basic parsing operations by parallelization. It is implemented in C and distributed as compiled code.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /your/installation/dir
cd /your/installation/dir
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/BitPar/BitPar.tar.gz
tar xzf BitPar.tar.gz 
cd BitPar/src
make
cd ../..
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

You will also need the parsing model for German which was trained on the Tiger treebank:






\begin{SaveVerbatim}{myverb}
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/BitPar/GermanParser.tar.gz
tar xzf GermanParser.tar.gz
cd GermanParser/src
make
cd ../..
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

There is also an English parsing model.



\subsubsection*{
\label{external-tools.texntoc17}LoPar (German)}


LoPar\footnote{\sf http://www.ims.uni-stuttgart.de/tcl/SOFTWARE/LoPar.html} is an implementation of a parser for head-lexicalized probabilistic context-free grammars, which can be also used for morphological analysis. The program is distributed without source code.



Installation:






\begin{SaveVerbatim}{myverb}
mkdir /my/installation/dir
cd /my/installation/dir
wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/LoPar/lopar-3.0.linux.tar.gz
tar xzf lopar-3.0.linux.tar.gz
cd LoPar-3.0
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
\label{external-tools.texntoc18}Berkeley Parser}


The Berkeley is a phrase structure grammar parser implemented in Java and distributed open source. Models are provided for English, Bugarian, Arabic, Chinese, French, German.



{\sf http://code.google.com/p/berkeleyparser/}



\subsection{
\label{external-tools.texntoc19}Other Open Source Machine Translation Systems}


\subsubsection*{
\label{external-tools.texntoc20}Joshua}


Joshua\footnote{\sf http://joshua.sourceforge.net/Joshua/Welcome.html} is a machine translation decoder for hierarchical models. Joshua development is centered at the Center for Language and Speech Processing at the Johns Hopkins University in Baltimore, Maryland. It is implemented in Java.



\subsubsection*{
\label{external-tools.texntoc21}cdec}


Cdec\footnote{\sf http://cdec-decoder.org/} is a decoder, aligner, and learning framework for statistical machine translation and other structured prediction models written by Chris Dyer in the University of Maryland Department of Linguistics. It is written in C++. 



\subsubsection*{
\label{external-tools.texntoc22}Apertium}


Apertium\footnote{\sf http://www.apertium.org/} is an open source rule-based machine translation (RBMT) system, maintained principally by the University of Alicante and Prompsit Engineering.



\subsubsection*{
\label{external-tools.texntoc23}Docent}


Docent\footnote{\sf https://github.com/chardmeier/docent/wiki} is a decoder for phrase-based SMT that treats complete documents, rather than single sentences, as translation units and permits the inclusion of features with cross-sentence dependencies. It is developed by Christian Hardmeier and implemented in C++



\subsubsection*{
\label{external-tools.texntoc24}Phrasal }


Phrase-based SMT toolkit written in Java. {\sf http://www-nlp.stanford.edu/wiki/Software/Phrasal2}



\subsection{
\label{external-tools.texntoc25}Other Translation Tools}


\subsubsection*{
\label{external-tools.texntoc26}COSTA MT Evaluation Tool}


COSTA MT Evaluation Tool\footnote{\sf https://code.google.com/p/costa-mt-evaluation-tool/} is an open-source Java program that can be used to evaluate manually the quality of the MT output. It is simple in use, designed to allow MT potential users and developers to analyse their engines using a friendly environment. It enables the ranking of the quality of MT output segment-by-segment for a particular language pair.



\subsubsection*{
\label{external-tools.texntoc27}Appraise}


Appraise\footnote{\sf https://github.com/cfedermann/Appraise} is an open-source tool for manual evaluation of Machine Translation output. Appraise allows to collect human judgments on translation output, implementing annotation tasks such as translation quality checking, ranking of translations, error classification, and manual post-editing. It is used in the ACL WMT evaluation campaign\footnote{\sf http://www.statmt.org/wmt13/}.



\subsubsection*{
\label{external-tools.texntoc28}Indic NLP Library}


Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.



The library provides the following functionalities:

\begin{SaveVerbatim}{myverb}
Text Normalization
Transliteration
Tokenization
Morphological Analysis
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

{\sf https://github.com/anoopkunchukuttan/indic\_nlp\_library}







\begin{center}\textit{\small Subsection last  modified on July 05, 2017, at 08:45 AM}\end{center}
